Re: Retrieve hadoop conf object from Python API
To close the loop.
This should work:
sc._jsc.hadoopConfiguration
See this method in JavaSparkContext :
def hadoopConfiguration(): Configuration = {
sc.hadoopConfiguration
On Tue, Jun 30, 2015 at 5:52 PM, Ted Yu yuzhih...@gmail.com wrote:
Minor correction:
It should be sc._jsc
Cheers
On Tue, Jun 30, 2015 at 4:23 PM, ayan guha guha.a...@gmail.com wrote:
There is a sc._jsc_ which you can access to get/set hadoop conf.
On Wed, Jul 1, 2015 at 7:41 AM, Richard Ding pigu...@gmail.com wrote:
Hi,
I noticed that, in Scala API, one can call hadoopConfiguration on
SparkContext to retrieve the hadoop configuration object which is very
handy in modifying certain hadoop properties at runtime. But there is no
corresponding method in Python API.
Will this method be added to Python API in a later release? In the mean
time, what will be the work around (of setting hadoop properties at
runtime) using Python API?
Thanks,
Richard
--
Best Regards,
Ayan Guha
Re: Retrieve hadoop conf object from Python API
Minor correction: It should be sc._jsc Cheers On Tue, Jun 30, 2015 at 4:23 PM, ayan guha guha.a...@gmail.com wrote: There is a sc._jsc_ which you can access to get/set hadoop conf. On Wed, Jul 1, 2015 at 7:41 AM, Richard Ding pigu...@gmail.com wrote: Hi, I noticed that, in Scala API, one can call hadoopConfiguration on SparkContext to retrieve the hadoop configuration object which is very handy in modifying certain hadoop properties at runtime. But there is no corresponding method in Python API. Will this method be added to Python API in a later release? In the mean time, what will be the work around (of setting hadoop properties at runtime) using Python API? Thanks, Richard -- Best Regards, Ayan Guha
Re: Retrieve hadoop conf object from Python API
There is a sc._jsc_ which you can access to get/set hadoop conf. On Wed, Jul 1, 2015 at 7:41 AM, Richard Ding pigu...@gmail.com wrote: Hi, I noticed that, in Scala API, one can call hadoopConfiguration on SparkContext to retrieve the hadoop configuration object which is very handy in modifying certain hadoop properties at runtime. But there is no corresponding method in Python API. Will this method be added to Python API in a later release? In the mean time, what will be the work around (of setting hadoop properties at runtime) using Python API? Thanks, Richard -- Best Regards, Ayan Guha
