Hi JL, Markus.
Any configuration properties (even though they're not listed in GUI by
default) start with 'spark.' can be added inside interpreter menu and
they're consumed by SparkContext.
About reading exsiting spark-defaults.conf, i think quickest way to do is
making little helper function in zeppelin-env.sh to read the configuration.
for example, if you place following code in your zeppelin-env.sh, it'll
read spark-defaults.conf and set properties.
function readSparkConf() {
> SPARK_CONF_PATH="${1}"
> echo "Reading ${SPARK_CONF_PATH}"
> while read line; do
> echo "${line}" | grep -e "^spark[.]" > /dev/null
> if [ $? -ne 0 ]; then
> # skip the line not started with 'spark.'
> continue;
> fi
> SPARK_CONF_KEY=`echo "${line}" | sed -e 's/\(^spark[^ ]*\)[
> \t]*\(.*\)/\1/g'`
> SPARK_CONF_VALUE=`echo "${line}" | sed -e 's/\(^spark[^ ]*\)[
> \t]*\(.*\)/\2/g'`
> export ZEPPELIN_JAVA_OPTS+="
> -D${SPARK_CONF_KEY}=\"${SPARK_CONF_VALUE}\""
> done < ${SPARK_CONF_PATH}
> }
> readSparkConf [YOUR_SPARK_HOME]/conf/spark-defaults.conf
Thanks,
moon
On Mon, Feb 23, 2015 at 12:56 PM, M. Dale <[email protected]> wrote:
> JL and Moon,
> Thank you for your responses below! I would second JL's observation.
> Creating a new interpreter via the GUI allows this customization but it
> would be nice to be able to just point at a configuration file (the same
> configuration file used to run spark-shell) and maybe add that to the
> default interpreter.
>
> Thank you,
> Markus
>
>
> On 02/22/2015 10:09 PM, Jongyoul Lee wrote:
>
> Hi Markus,
>
> I also cannot find to pass spark configurations to zeppelin easily, but
> you can do it with JAVA_OPTS. In your case,
> JAVA_OPTS="-Dspark.serializer=org.apache.spark.serializer.KryoSerializer
> -Dspark.kryo.registrator=com.uebercomputing.mailrecord.MailRecordRegistrator"
>
> Hi Moon,
> Zeppelin needs to read spark-default.conf, adds a configuration option to
> set spark-default.conf, or allow set spark.* to zeppelin-site.xml. Please
> give us your opinion.
>
> Regards,
> JL
>
> On Mon, Feb 23, 2015 at 10:24 AM, moon soo Lee <[email protected]>
> wrote:
>
>> Hi Markus,
>>
>> I think following directories
>>
>> conf/ (basic configurations, interpreter configurations)
>> notebook/ (notebooks)
>> interpreter/ (if you have any 3rd party interpreter installed)
>>
>> are things to take care when move to next Zeppelin version.
>>
>> Thanks
>> moon
>>
>>
>> On Fri, Feb 20, 2015 at 12:57 AM, M. Dale <[email protected]> wrote:
>>
>>> Anthony,
>>> That did the trick! Thank you so much for your quick reply.
>>>
>>> It seems that conf/interpreter.json got changed and now saves my new
>>> interpreter. When I move to the next version of Zeppelin, is that what I
>>> need to copy to keep that interpreter as an option?
>>>
>>> Thanks again for your help,
>>> Markus
>>>
>>>
>>> On 02/19/2015 10:46 AM, Anthony Corbacho wrote:
>>>
>>> hello Markus,
>>>
>>> in Zeppelin you can configure those option through the interpreters.
>>>
>>> basically, you will have to create a new interpreter and set the
>>> serialization options (you will see that you can add custom options to
>>> spark interpreter) and then activate this interpreter in your notebook.
>>>
>>> hope it help.
>>> On Feb 20, 2015 12:41 AM, "M. Dale" <[email protected]> wrote:
>>>
>>>> I would like to use Spark with Kryo serialization.
>>>>
>>>> From the command line spark-shell I would add:
>>>> --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
>>>> --conf
>>>> spark.kryo.registrator=com.uebercomputing.mailrecord.MailRecordRegistrator
>>>> \
>>>>
>>>> Or I can add those to conf/spark-defaults.conf:
>>>> spark.serializer=org.apache.spark.serializer.KryoSerializer
>>>>
>>>> spark.kryo.registrator=com.uebercomputing.mailrecord.MailRecordRegistrator
>>>>
>>>> How can I set these properties for the Zeppelin-provided spark context?
>>>>
>>>> Thanks for your help,
>>>> Markus
>>>>
>>>
>>>
>>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
>
>
