Re: How do I set spark conf parameters?

[Meisam Fathi]

On a more general note, with or without Livy, Spark configurations should
be set before creating the Spark contex (sc). Setting configs after sc is
created has no effect.

Thanks,
Meisam

On Wed, Jan 31, 2018 at 3:14 AM Stefan Miklosovic 
wrote:

> livyClient = new LivyClientBuilder()
>   .setAll(withSparkProperties())
>   .setURI(new URI(LIVY_URI))
>   .build()
>
>
>
> On Wed, Jan 31, 2018 at 12:13 PM, Stefan Miklosovic 
> wrote:
>
>> Yes, I am using setAll(Properties).
>>
>> On Wed, Jan 31, 2018 at 11:45 AM, kant kodali  wrote:
>>
>>> Sorry. I found an answer online.It should be something like this
>>>
>>> new LivyClientBuilder().setConf("spark.es.index.auto.create", 
>>> "true").setConf("spark.cassandra.connection.host", "127.0.0.01").build();
>>>
>>>
>>> On Wed, Jan 31, 2018 at 1:16 AM, kant kodali  wrote:
>>>
>>>> Hi All,
>>>>
>>>>
>>>> How do I set Spark Conf Parameters ? The below doesnt seem to get
>>>> picked up? If so, how can I change my program such that it can pick it up.
>>>> I am not seeing a way if sparkcontext is already created?
>>>>
>>>> public String call(JobContext ctx) throws Exception {
>>>> ctx.sc().setLogLevel("INFO");
>>>> ctx.sc().getConf()
>>>> .set("spark.cassandra.connection.host", 
>>>> config.getString("cassandra.host"))
>>>> .set("spark.cassandra.auth.username", 
>>>> config.getString("cassandra.user"))
>>>> .set("spark.cassandra.auth.password", 
>>>> config.getString("cassandra.pass"))
>>>> .set("es.index.auto.create", "true");
>>>>
>>>> }
>>>>
>>>>
>>>> Thanks!
>>>>
>>>>
>>>
>>
>>
>> --
>> Stefan Miklosovic
>>
>
>
>
> --
> Stefan Miklosovic
>

Re: How do I set spark conf parameters?

[Stefan Miklosovic]

Yes, I am using setAll(Properties).

On Wed, Jan 31, 2018 at 11:45 AM, kant kodali  wrote:

> Sorry. I found an answer online.It should be something like this
>
> new LivyClientBuilder().setConf("spark.es.index.auto.create", 
> "true").setConf("spark.cassandra.connection.host", "127.0.0.01").build();
>
>
> On Wed, Jan 31, 2018 at 1:16 AM, kant kodali  wrote:
>
>> Hi All,
>>
>>
>> How do I set Spark Conf Parameters ? The below doesnt seem to get picked
>> up? If so, how can I change my program such that it can pick it up. I am
>> not seeing a way if sparkcontext is already created?
>>
>> public String call(JobContext ctx) throws Exception {
>> ctx.sc().setLogLevel("INFO");
>> ctx.sc().getConf()
>> .set("spark.cassandra.connection.host", 
>> config.getString("cassandra.host"))
>> .set("spark.cassandra.auth.username", 
>> config.getString("cassandra.user"))
>> .set("spark.cassandra.auth.password", 
>> config.getString("cassandra.pass"))
>> .set("es.index.auto.create", "true");
>>
>> }
>>
>>
>> Thanks!
>>
>>
>


-- 
Stefan Miklosovic

Re: How do I set spark conf parameters?

[Stefan Miklosovic]

livyClient = new LivyClientBuilder()
  .setAll(withSparkProperties())
  .setURI(new URI(LIVY_URI))
  .build()



On Wed, Jan 31, 2018 at 12:13 PM, Stefan Miklosovic 
wrote:

> Yes, I am using setAll(Properties).
>
> On Wed, Jan 31, 2018 at 11:45 AM, kant kodali  wrote:
>
>> Sorry. I found an answer online.It should be something like this
>>
>> new LivyClientBuilder().setConf("spark.es.index.auto.create", 
>> "true").setConf("spark.cassandra.connection.host", "127.0.0.01").build();
>>
>>
>> On Wed, Jan 31, 2018 at 1:16 AM, kant kodali  wrote:
>>
>>> Hi All,
>>>
>>>
>>> How do I set Spark Conf Parameters ? The below doesnt seem to get picked
>>> up? If so, how can I change my program such that it can pick it up. I am
>>> not seeing a way if sparkcontext is already created?
>>>
>>> public String call(JobContext ctx) throws Exception {
>>> ctx.sc().setLogLevel("INFO");
>>> ctx.sc().getConf()
>>> .set("spark.cassandra.connection.host", 
>>> config.getString("cassandra.host"))
>>> .set("spark.cassandra.auth.username", 
>>> config.getString("cassandra.user"))
>>> .set("spark.cassandra.auth.password", 
>>> config.getString("cassandra.pass"))
>>> .set("es.index.auto.create", "true");
>>>
>>> }
>>>
>>>
>>> Thanks!
>>>
>>>
>>
>
>
> --
> Stefan Miklosovic
>



-- 
Stefan Miklosovic

Re: How do I set spark conf parameters?

[kant kodali]

Sorry. I found an answer online.It should be something like this

new LivyClientBuilder().setConf("spark.es.index.auto.create",
"true").setConf("spark.cassandra.connection.host",
"127.0.0.01").build();


On Wed, Jan 31, 2018 at 1:16 AM, kant kodali  wrote:

> Hi All,
>
>
> How do I set Spark Conf Parameters ? The below doesnt seem to get picked
> up? If so, how can I change my program such that it can pick it up. I am
> not seeing a way if sparkcontext is already created?
>
> public String call(JobContext ctx) throws Exception {
> ctx.sc().setLogLevel("INFO");
> ctx.sc().getConf()
> .set("spark.cassandra.connection.host", 
> config.getString("cassandra.host"))
> .set("spark.cassandra.auth.username", 
> config.getString("cassandra.user"))
> .set("spark.cassandra.auth.password", 
> config.getString("cassandra.pass"))
> .set("es.index.auto.create", "true");
>
> }
>
>
> Thanks!
>
>

How do I set spark conf parameters?

[kant kodali]

Hi All,


How do I set Spark Conf Parameters ? The below doesnt seem to get picked
up? If so, how can I change my program such that it can pick it up. I am
not seeing a way if sparkcontext is already created?

public String call(JobContext ctx) throws Exception {
ctx.sc().setLogLevel("INFO");
ctx.sc().getConf()
.set("spark.cassandra.connection.host",
config.getString("cassandra.host"))
.set("spark.cassandra.auth.username",
config.getString("cassandra.user"))
.set("spark.cassandra.auth.password",
config.getString("cassandra.pass"))
.set("es.index.auto.create", "true");

}


Thanks!