Can you give the output of “printSchema” ?
> On 26 Mar 2018, at 22:39, Nikhil Goyal wrote:
>
> Hi guys,
>
> I have a Map[(String, String), Double] as one of my columns. Using
> input.getAs[Map[(String, String), Double]](0)
> throws exception: Caused by: java.lang.ClassCastException:
> org.ap
Thanks
> On 26 Mar 2018, at 22:09, Marcelo Vanzin wrote:
>
> On Mon, Mar 26, 2018 at 1:08 PM, Gauthier Feuillen
> wrote:
>> Is there a way to change this value without changing yarn-site.xml ?
>
> No. Local dirs are defined by the NodeManager, and Sp
Hi
I am trying to change the spark.local.dir property. I am running spark on yarn
and have already tried the following properties:
export LOCAL_DIRS=
spark.yarn.appMasterEnv.LOCAL_DIRS=
spark.yarn.appMasterEnv.SPARK_LOCAL_DIRS=
spark.yarn.nodemanager.local-dirs=/
spark.local.dir=
But still it