yes it does. i only see this problem in zeppelin. On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <hss...@nflabs.com> wrote:
> Hi Jonathan. > It's not easy to build the test environments but I am working on this. > I have question for you. > Does your code working well on spark-shell in the spark.authenticate mode? > > 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy < > jonathan.esterh...@gmail.com>: > >> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm >> quite stumped. >> >> thx. >> >> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <hss...@nflabs.com> >> wrote: >> >>> Thank you. >>> Let me try. >>> >>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy < >>> jonathan.esterh...@gmail.com>: >>> >>>> Hyung, >>>> >>>> Yes, here they are. >>>> >>>> zeppelin-env.sh: >>>> >>>> export ZEPPELIN_PORT=8890 >>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf >>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin >>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin >>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid >>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook >>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps >>>> export MASTER=yarn-client >>>> export SPARK_HOME=/usr/lib/spark >>>> export HADOOP_CONF_DIR=/etc/hadoop/conf >>>> export >>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/* >>>> :/usr/share/aws/emr/emrfs/auxlib/*" >>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0 >>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket >>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin >>>> export >>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo >>>> >>>> spark-defaults.conf: >>>> >>>> spark.master yarn >>>> spark.driver.extraClassPath >>>> >>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf >>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* >>>> spark.driver.extraLibraryPath >>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native >>>> spark.executor.extraClassPath >>>> >>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf >>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* >>>> spark.executor.extraLibraryPath >>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native >>>> spark.eventLog.enabled true >>>> spark.eventLog.dir hdfs:///var/log/spark/apps >>>> spark.history.fs.logDirectory hdfs:///var/log/spark/apps >>>> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080 >>>> spark.history.ui.port 18080 >>>> spark.shuffle.service.enabled true >>>> spark.driver.extraJavaOptions >>>> -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties >>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 >>>> -XX:MaxHeapFreeRatio=70 >>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M >>>> -XX:OnOutOfMemoryError='kill -9 %p' >>>> spark.dynamicAllocation.enabled true >>>> spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails >>>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC >>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM >>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p' >>>> spark.executor.memory 8640m >>>> spark.executor.cores 7 >>>> spark.authenticate.enableSaslEncryption true >>>> spark.driver.memory 1g >>>> spark.network.sasl.serverAlwaysEncrypt true >>>> spark.driver.cores 1 >>>> spark.ssl.protocol TLSv1.2 >>>> spark.ssl.keyStorePassword password >>>> spark.yarn.maxAppAttempts 1 >>>> spark.ssl.keyStore /etc/emr/security/keystore.jks >>>> spark.authenticate true >>>> spark.ssl.keyPassword password >>>> spark.ssl.enabled true >>>> spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_CBC_SHA >>>> spark.ssl.trustStore /etc/emr/security/truststore.jks >>>> spark.authenticate.secret secret >>>> spark.ssl.trustStorePassword password >>>> >>>> >>>> >>>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim <hss...@nflabs.com> >>>> wrote: >>>> >>>>> Hi. >>>>> Could you share your conf/zeppelin-env.sh and spark-defaults.conf ? >>>>> >>>>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy < >>>>> jonathan.esterh...@gmail.com>: >>>>> >>>>>> I am having trouble using zeppelin in a spark cluster that has spark >>>>>> node authentication turned on (e.g. with spark.authenticate=true, >>>>>> spark.authenticate.secret=...) >>>>>> >>>>>> Notebook code that calls built-in spark functions (or other things on >>>>>> executor classpath) work fine, but functions defined in the notebook >>>>>> (anonymous or named) throw ClassNotFoundExceptions when called from an >>>>>> executor. >>>>>> >>>>>> For example, this code works: >>>>>> >>>>>> val rdd = sc.textFile("hdfs://my-text-file") >>>>>> rdd.take(1).foreach(println) >>>>>> >>>>>> rdd.saveAsTextFile("hdfs:///my-other-text-file") >>>>>> >>>>>> but code like this... >>>>>> >>>>>> rdd.filter(_.contains("my data")) >>>>>> >>>>>> fails with >>>>>> >>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 >>>>>> at >>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>> at java.lang.Class.forName0(Native Method) >>>>>> at java.lang.Class.forName(Class.java:348) >>>>>> ... >>>>>> >>>>>> >>>>>> I get the same kind of error if the filter function is defined as a >>>>>> named function in the notebook, or as a member of singleton object >>>>>> defined >>>>>> in the notebook. >>>>>> >>>>>> When I look at the executor's log output, I see this error: >>>>>> >>>>>> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to check >>>>>> existence of class >>>>>> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on REPL >>>>>> class server at https://172.30.54.30:34980 >>>>>> java.lang.NullPointerException >>>>>> at >>>>>> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113) >>>>>> at >>>>>> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146) >>>>>> at >>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>> at java.lang.Class.forName0(Native Method) >>>>>> at java.lang.Class.forName(Class.java:348) >>>>>> >>>>>> ... >>>>>> >>>>>> >>>>>> If I disable spark authentication, everything works as expected. I am >>>>>> running zeppelin 0.5.6 on spark 1.6.1 with yarn. >>>>>> >>>>>> Has anyone been able to get zeppelin working with spark >>>>>> authentication? >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >