Thanks for the response,
I'm not sure what do you mean, it exactly what I tried and failed.
As I wrote above, 'hc' is actually different name to sqlc (that is
different name to z.sqlContext).
I get the same results.
On Mon, Jun 29, 2015 at 2:12 PM, Mina Lee <mina...@nflabs.com> wrote:
> Hi Ophir,
>
> Can you try below?
>
> def getNum(): Int = {
> 100
> }
> sqlc.udf.register("getNum", getNum _)
> sqlc.sql("select getNum() from filteredNc limit 1").show
>
> FYI sqlContext(==sqlc) is internally created by Zeppelin
> and use hiveContext as sqlContext by default.
> (If you did not change useHiveContext to be "false" in interpreter menu.)
>
> Hope it helps.
>
> On Mon, Jun 29, 2015 at 7:55 PM, Ophir Cohen <oph...@gmail.com> wrote:
>
>> Guys?
>> Somebody?
>> Can it be that Zeppelin does not support UDFs?
>>
>> On Sun, Jun 28, 2015 at 11:53 AM, Ophir Cohen <oph...@gmail.com> wrote:
>>
>>> Hi Guys,
>>> One more problem I have encountered using Zeppelin.
>>> Using Spark 1.3.1 on Yarn Hadoop 2.4
>>>
>>> I'm trying to create and use UDF (hc == z.sqlContext == HiveContext):
>>> 1. Create and register the UDF:
>>> def getNum(): Int = {
>>> 100
>>> }
>>>
>>> hc.udf.register("getNum",getNum _)
>>> 2. And I try to use on exist table:
>>> %sql select getNum() from filteredNc limit 1
>>>
>>> Or:
>>> 3. Trying using direct hc:
>>> hc.sql("select getNum() from filteredNc limit 1").collect
>>>
>>> Both of them yield with
>>> *"java.lang.ClassNotFoundException:
>>> org.apache.zeppelin.spark.ZeppelinContext"*
>>> (see below the full exception).
>>>
>>> And my questions is:
>>> 1. Can it be that ZeppelinContext is not available on Spark nodes?
>>> 2. Why it need ZeppelinContext anyway? Why it's relevant?
>>>
>>> The exception:
>>> WARN [2015-06-28 08:43:53,850] ({task-result-getter-0}
>>> Logging.scala[logWarning]:71) - Lost task 0.2 in stage 23.0 (TID 1626,
>>> ip-10-216-204-246.ec2.internal): java.lang.NoClassDefFoundError:
>>> Lorg/apache/zeppelin/spark/ZeppelinContext;
>>> at java.lang.Class.getDeclaredFields0(Native Method)
>>> at java.lang.Class.privateGetDeclaredFields(Class.java:2499)
>>> at java.lang.Class.getDeclaredField(Class.java:1951)
>>> at
>>> java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1659)
>>>
>>> <Many more of ObjectStreamClass lines of exception>
>>>
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.zeppelin.spark.ZeppelinContext
>>> at
>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:69)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>> ... 103 more
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.zeppelin.spark.ZeppelinContext
>>> at java.lang.ClassLoader.findClass(ClassLoader.java:531)
>>> at
>>> org.apache.spark.util.ParentClassLoader.findClass(ParentClassLoader.scala:26)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>> at
>>> org.apache.spark.util.ParentClassLoader.loadClass(ParentClassLoader.scala:34)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>> at
>>> org.apache.spark.util.ParentClassLoader.loadClass(ParentClassLoader.scala:30)
>>> at
>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:64)
>>> ... 105 more
>>>
>>
>>
>
