Re: Spark Shell doesnt seem to use spark workers but Spark Submit does.

2016-11-23 Thread kant kodali
somehow the table scan to do the count of billion rows in Cassandra is not being done in parallel. On Wed, Nov 23, 2016 at 12:45 PM, kant kodali wrote: > Hi All, > > > Spark Shell doesnt seem to use spark workers but Spark Submit does. I had > the workers ips listed under conf/sla

Spark Shell doesnt seem to use spark workers but Spark Submit does.

2016-11-23 Thread kant kodali
Hi All, Spark Shell doesnt seem to use spark workers but Spark Submit does. I had the workers ips listed under conf/slaves file. I am trying to count number of rows in Cassandra using spark-shell so I do the following on spark master val df = spark.sql("SELECT test from hello") /