Re: Installing Distributed apache spark cluster with Cluster mode on Docker

Can you share your Dockerfile (not all but gist of it) and instructions how you do it and what you actually run to get that message? I have just pushed my local repo to Github where I have created an example of Spark on Docker some time ago. Please take a look and compare what you are doing. http

Re: Installing Distributed apache spark cluster with Cluster mode on Docker

Standalone mode already implies you are running on cluster (distributed) mode. i.e. it's one of 4 available cluster manager options. The difference is Standalone uses it's one resource manager rather than using YARN for example. If you are running docker on a single machine then you are limited to

Re: Installing Distributed apache spark cluster with Cluster mode on Docker

Hi Khalid Mammadov, Thank you for your response, Yes, I did, I built standalone apache spark cluster on docker containers. But I am looking for distributed spark cluster, Where spark workers are scalable and spark "deployment mode = cluster". Source url I used to built standalone apache spark c

Re: Installing Distributed apache spark cluster with Cluster mode on Docker

Hi, Have you checked out docs? https://spark.apache.org/docs/latest/spark-standalone.html Thanks, Khalid On Sat, Jul 24, 2021 at 1:45 PM Dinakar Chennubotla < chennu.bigd...@gmail.com> wrote: > Hi All, > > I am Dinakar, Hadoop admin, > could someone help me here, > > 1. I have a DEV-POC task to

Installing Distributed apache spark cluster with Cluster mode on Docker

Hi All, I am Dinakar, Hadoop admin, could someone help me here, 1. I have a DEV-POC task to do, 2. Need to Installing Distributed apache-spark cluster with Cluster mode on Docker containers. 3. with Scalable spark-worker containers. 4. we have a 9 node cluster with some other services or tools.

spark-submit --files causes spark context to fail in Kubernetes

Spark documentation refers to spark-sumit --files as --files FILES: Comma-separated list of files to be placed in the working directory of each executor. OK I have implemented this one for Kubernetes as per Spark doc as follows: