Re: how to use sc.hadoopConfiguration from pyspark

2015-11-23 Thread Eike von Seggern
2015-11-23 10:26 GMT+01:00 Tamas Szuromi : > Hello Eike, > > Thanks! Yes I'm using it with Hadoop 2.6 so I'll give a try to the 2.4 > build. > Have you tried it with 1.6 Snapshot or do you know JIRA tickets for this > missing libraries issues? I've not tried 1.6. https://issues.apache.org/jira/br

Re: how to use sc.hadoopConfiguration from pyspark

2015-11-23 Thread Tamas Szuromi
Hello Eike, Thanks! Yes I'm using it with Hadoop 2.6 so I'll give a try to the 2.4 build. Have you tried it with 1.6 Snapshot or do you know JIRA tickets for this missing libraries issues? Tamas On 23 November 2015 at 10:21, Eike von Seggern wrote: > Hello Tamas, > > 2015-11-20 17:23 GMT+0

Re: how to use sc.hadoopConfiguration from pyspark

2015-11-23 Thread Eike von Seggern
Hello Tamas, 2015-11-20 17:23 GMT+01:00 Tamas Szuromi : > > Hello, > > I've just wanted to use sc._jsc.hadoopConfiguration().set('key','value') in > pyspark 1.5.2 but I got set method not exists error. For me it's working with Spark 1.5.2 binary distribution built against Hadoop 2.4 (spark-1.5.

how to use sc.hadoopConfiguration from pyspark

2015-11-20 Thread Tamas Szuromi
Hello, I've just wanted to use sc._jsc.hadoopConfiguration().set('key','value') in pyspark 1.5.2 but I got set method not exists error. Are there anyone who know a workaround to set some hdfs related properties like dfs.blocksize? Thanks in advance! Tamas