Spark on K8s with Romana

So, we've run into something interesting. In our case, we've got some proprietary networking HW which is very feature limited in the TCP/IP space, so using Romana, executors can't seem to find the driver using the hostname lookup method it's attempting. Is there any way to make it use IP? Thanks,

Spark on K8s - using files fetched by init-container?

This is probably stupid user error, but I can't for the life of me figure out how to access the files that are staged by the init-container. I'm trying to run the SparkR example data-manipulation.R which requires the path to its datafile. I supply the hdfs location via --files and then the full

Re: Spark on K8s - using files fetched by init-container?

v` > instead. The 'file://' scheme must be explicitly used as it seems it > defaults to 'hdfs' in your setup. > > On Mon, Feb 26, 2018 at 12:57 PM, Jenna Hoole <jenna.ho...@gmail.com> > wrote: > >> Thank you for the quick response! However, I'm still having problems. >>

Spark on K8s resource staging server timeout

So I'm running into an issue with my resource staging server that's producing a stacktrace like Issue 342 , but I don't think for the same reasons. What's happening is that every time after I start up a resource staging server, the first job

Re: Spark on K8s resource staging server timeout

) .writeTimeout(120, TimeUnit.SECONDS) .readTimeout(120, TimeUnit.SECONDS) -Jenna On Tue, Mar 27, 2018 at 10:48 AM, Jenna Hoole <jenna.ho...@gmail.com> wrote: > So I'm running into an issue with my resource staging server that's > producing a stacktrace like Issue 342 > <

Re: Spark on K8s resource staging server timeout

the official > release we should probably create a mechanism that’s more resilient. Using > a single HTTP server isn’t ideal – would ideally like something that’s > highly available, replicated, etc. > > > > -Matt Cheah > > > > *From: *Jenna Hoole <jenna