[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-494734965 @erikerlandson from what I see there is no more activity, could you merge please? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-494104863 @erikerlandson @vanzin are you ok with the current status of things? Should it be merged? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-493931364 @srowen @erikerlandson gentle ping This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-493568264 @srowen I fixed the two pending comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-492988663 @erikerlandson @srowen I added support for random dirs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-492987168 jenkins test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-492779301 jenkins test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-492290588 @srowen ok will do that and if someone can always change the code with a new PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-492182627 @srowen one reason I dont want to delete the subdir it is because user provides it and may want to re-use the contents of it, because in a consecutive submission he may dont want to re-upload the jar. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-492000853 @srowen thoughts? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491546073 @srowen there a property that can be used by the user to set where he wants to upload the artifacts. By specifying the full path which may have subdirs things can be isolated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491523244 @srowen I also felt missing the counterargument. @vanzin could you clarify what is the issue here so we can move forward? I also hope @erikerlandson and @liyinan926 to comment here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491015268 @vanzin can you also focus on my argument I never used such a word btw. It is not the first time things are not constructive. Maybe it shows something anyway im out too. If the other committers dont want to deal with the situation I will do the changes because the added value of the PR goes beyond this minor issue, unfortunately this is not noticable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491008961 Btw using strong language to describe a choice and which creates just impressions eg. terrible is not a constructive and I would expect better from committers Disappointed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491006649 Should spark manage s3 folders accounts etc? This is optimized for the app dev cycle where I do a change and my updated jar is uploaded and is then overwritten. Im not also not comfortable with the attitude I deal with but I guess this how things has always been. Anyway im not convinced with the argument. Btw Im not sure Spark deletes all files it generates like checkpoints because they can bw reused and only the user knows when. Still uploaded files are not generated by Spark they come from the user. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-490870511 @vanzin this is not a big its a design choice. If you want separate uploads define a separate path. Your OS does not provide that convenience for you why should i? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-490191825 @srowen @vanzin should we merge it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-487606840 @vanzin gentle ping. I think its ready. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-487442281 jenkins test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client
dependencies with a Hadoop Compatible File System
URL: https://github.com/apache/spark/pull/23546#issuecomment-487419487
@shaneknapp there is a weird error:
```
2019-04-28 22:01:02 INFO MicroBatchExecution:54 - Streaming query made
progress: {
"id" : "6a171773-7f21-478d-9a4b-1fe2fe47f311",
"runId" : "6ed1ac89-76ca-4cdb-b536-7f68968423eb",
"name" : null,
"timestamp" : "2019-04-28T22:00:55.748Z",
"batchId" : 2,
"numInputRows" : 47614,
"inputRowsPerSecond" : 2513.407939189189,
"processedRowsPerSecond" : 7347.839506172839,
"durationMs" : {
"addBatch" : 1393,
"getBatch" : 1,
"getEndOffset" : 0,
"queryPlanning" : 62,
"setOffsetRange" : 7,
"triggerExecution" : 6480,
"walCommit" : 3099
},
"stateOperators" : [ ],
"sources" : [ {
"description" : "KafkaV2[Subscribe[ss-in-topic]]",
"startOffset" : {
"ss-in-topic" : {
"2" : 19020,
"5" : 19363,
"4" : 19299,
"1" : 19147,
"3" : 19247,
"0" : 19517
}
},
"endOffset" : {
"ss-in-topic" : {
"2" : 26928,
"5" : 27398,
"4" : 27104,
"1" : 27163,
"3" : 27108,
"0" : 27506
}
},
"numInputRows" : 47614,
"inputRowsPerSecond" : 2513.407939189189,
"processedRowsPerSecond" : 7347.839506172839
} ],
"sink" : {
"description" :
"org.apache.spark.sql.kafka010.KafkaSourceProvider@1672d865"
}
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-487418537 jenkins test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-48716 @erikerlandson gentle ping :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client
dependencies with a Hadoop Compatible File System
URL: https://github.com/apache/spark/pull/23546#issuecomment-484733537
@shaneknapp
```
> devtools::install_version('testthat', version = '1.0.2',
repos='http://cran.us.r-project.org')
Error in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck =
vI[[j]]) :
there is no package called 'processx'
Calls: :: ... tryCatch -> tryCatchList -> tryCatchOne ->
Execution halted
```
fyi I did a rebase to get updates from master.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-484654875 @shaneknapp do you know why checks fail here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-484644274 @srowen I fixed most stuff, for things not modified there is a comment. Please have a look. Yes the concern is that it touches Spark Submit but things should not be that fragile to touch IMHO. We should slowly refactor it so we are more confident about it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-483434764 @erikerlandson gentle ping. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-482549358 @erikerlandson @liyinan926 @vanzin PR has been updated with the addition of integration tests. As stated in the description [ceph nano](https://github.com/ceph/cn) is used based on ceph images. If you want to check what is being written just `kubectl port-forward ceph-nano-0 -n 5001:5001`. Here is a view of the bucket created during test run:  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-477112560 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-474880219 WIP This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-472035217 @erikerlandson I think rook.io with ceph might be a great option for the tests here instead of minio due the restrictions the latter has: https://redhatstorage.redhat.com/2018/06/25/why-spark-on-ceph-part-1-of-3/ https://rook.io/ https://radanalytics.io/examples/ceph-source-example This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
