Re: Spark Sql - Missing Jar ? json_tuple NoClassDefFoundError

[Cheng Lian]

Filed https://issues.apache.org/jira/browse/SPARK-6708 to track this.

Cheng

On 4/4/15 10:21 PM, Cheng Lian wrote:

I think this is a bug of Spark SQL dates back to at least 1.1.0.

The json_tuple function is implemented as 
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFJSONTuple. The 
ClassNotFoundException should complain with the class name rather than 
the UDTF function name.


The problematic line should be this one 
. 
HiveFunctionWrapper expects the full qualified class name of the UDTF 
class that implements the function, but we pass in the function name.


Thanks for reporting this!

Cheng

On 4/2/15 3:19 AM, Todd Nist wrote:


I have a feeling I’m missing a Jar that provides the support or could 
this may be related to 
https://issues.apache.org/jira/browse/SPARK-5792. If it is a Jar 
where would I find that ? I would have thought in the $HIVE/lib 
folder, but not sure which jar contains it.


Error:

|Create  MetricTemporary  Table  for  querying
15/04/01  14:41:44  INFO HiveMetaStore:0: Opening raw storewith  implemenation 
class:org.apache.hadoop.hive.metastore.ObjectStore
15/04/01  14:41:44  INFO ObjectStore: ObjectStore, initialize called
15/04/01  14:41:45  INFO Persistence: Property 
hive.metastore.integral.jdo.pushdownunknown  - will be ignored
15/04/01  14:41:45  INFO Persistence: Property datanucleus.cache.level2unknown  
- will be ignored
15/04/01  14:41:45  INFO BlockManager: Removing broadcast0
15/04/01  14:41:45  INFO BlockManager: Removing block broadcast_0
15/04/01  14:41:45  INFO MemoryStore: Block broadcast_0of  size  1272  
droppedfrom  memory (free278018571)
15/04/01  14:41:45  INFO BlockManager: Removing block broadcast_0_piece0
15/04/01  14:41:45  INFO MemoryStore: Block broadcast_0_piece0of  size  869  
droppedfrom  memory (free278019440)
15/04/01  14:41:45  INFO BlockManagerInfo: Removed broadcast_0_piece0on  
192.168.1.5:63230  in  memory (size:869.0  B, free:265.1  MB)
15/04/01  14:41:45  INFO BlockManagerMaster: Updated infoof  block 
broadcast_0_piece0
15/04/01  14:41:45  INFO BlockManagerInfo: Removed broadcast_0_piece0on  
192.168.1.5:63278  in  memory (size:869.0  B, free:530.0  MB)
15/04/01  14:41:45  INFO ContextCleaner: Cleaned broadcast0
15/04/01  14:41:46  INFO ObjectStore: Setting MetaStore object pin classeswith  
hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
15/04/01  14:41:46  INFO Datastore: The 
class"org.apache.hadoop.hive.metastore.model.MFieldSchema"  is  taggedas  
"embedded-only"  so doesnot  have its own datastoretable.
15/04/01  14:41:46  INFO Datastore: The 
class"org.apache.hadoop.hive.metastore.model.MOrder"  is  taggedas  
"embedded-only"  so doesnot  have its own datastoretable.
15/04/01  14:41:47  INFO Datastore: The 
class"org.apache.hadoop.hive.metastore.model.MFieldSchema"  is  taggedas  
"embedded-only"  so doesnot  have its own datastoretable.
15/04/01  14:41:47  INFO Datastore: The 
class"org.apache.hadoop.hive.metastore.model.MOrder"  is  taggedas  
"embedded-only"  so doesnot  have its own datastoretable.
15/04/01  14:41:47  INFO Query: Readingin  resultsfor  
query"org.datanucleus.store.rdbms.query.SQLQuery@0"  since theconnection  
usedis  closing
15/04/01  14:41:47  INFO ObjectStore: Initialized ObjectStore
15/04/01  14:41:47  INFO HiveMetaStore: Added admin rolein  metastore
15/04/01  14:41:47  INFO HiveMetaStore: Addedpublic  rolein  metastore
15/04/01  14:41:48  INFO HiveMetaStore:No  user  is  addedin  admin role, since 
configis  empty
15/04/01  14:41:48  INFO SessionState:No  Tezsession  requiredat  this point. 
hive.execution.engine=mr.
15/04/01  14:41:49  INFO ParseDriver: Parsing command:SELECT  path, name,value, 
v1.peValue, v1.peName
  FROM  metric
  lateralview  json_tuple(pathElements,'name','value') v1
as  peName, peValue
15/04/01  14:41:49  INFO ParseDriver: Parse Completed
Exception  in  thread"main"  java.lang.ClassNotFoundException: json_tuple
 at  java.net.URLClassLoader$1.run(URLClassLoader.java:372)
 at  java.net.URLClassLoader$1.run(URLClassLoader.java:361)
 at  java.security.AccessController.doPrivileged(Native Method)
 at  java.net.URLClassLoader.findClass(URLClassLoader.java:360)
 at  java.lang.ClassLoader.loadClass(ClassLoader.java:424)
 at  java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 at  
org.apache.spark.sql.hive.HiveFunctionWrapper.createFunction(Shim13.scala:141)
 at  
org.apache.spark.sql.hive.HiveGenericUdtf.function$lzycompute(hiveUdfs.scala:261)
 at  org.apache.spark.sql.hive.HiveGenericUdtf.function(hiveUdfs.scala:261)
 at  
org.apache.spark.sql.hive.HiveGenericUdtf.outputInspector$lzycompute(hiveUdfs.scala:267)
 at  
org.apache.spark.sql.hive.HiveGenericUdtf.outputInspector(hiveUdfs.scala:

Re: Spark Sql - Missing Jar ? json_tuple NoClassDefFoundError

[Cheng Lian]

I think this is a bug of Spark SQL dates back to at least 1.1.0.

The json_tuple function is implemented as 
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFJSONTuple. The 
ClassNotFoundException should complain with the class name rather than 
the UDTF function name.


The problematic line should be this one 
. 
HiveFunctionWrapper expects the full qualified class name of the UDTF 
class that implements the function, but we pass in the function name.


Thanks for reporting this!

Cheng

On 4/2/15 3:19 AM, Todd Nist wrote:


I have a feeling I’m missing a Jar that provides the support or could 
this may be related to 
https://issues.apache.org/jira/browse/SPARK-5792. If it is a Jar where 
would I find that ? I would have thought in the $HIVE/lib folder, but 
not sure which jar contains it.


Error:

|Create  MetricTemporary  Table  for  querying
15/04/01  14:41:44  INFO HiveMetaStore:0: Opening raw storewith  implemenation 
class:org.apache.hadoop.hive.metastore.ObjectStore
15/04/01  14:41:44  INFO ObjectStore: ObjectStore, initialize called
15/04/01  14:41:45  INFO Persistence: Property 
hive.metastore.integral.jdo.pushdownunknown  - will be ignored
15/04/01  14:41:45  INFO Persistence: Property datanucleus.cache.level2unknown  
- will be ignored
15/04/01  14:41:45  INFO BlockManager: Removing broadcast0
15/04/01  14:41:45  INFO BlockManager: Removing block broadcast_0
15/04/01  14:41:45  INFO MemoryStore: Block broadcast_0of  size  1272  
droppedfrom  memory (free278018571)
15/04/01  14:41:45  INFO BlockManager: Removing block broadcast_0_piece0
15/04/01  14:41:45  INFO MemoryStore: Block broadcast_0_piece0of  size  869  
droppedfrom  memory (free278019440)
15/04/01  14:41:45  INFO BlockManagerInfo: Removed broadcast_0_piece0on  
192.168.1.5:63230  in  memory (size:869.0  B, free:265.1  MB)
15/04/01  14:41:45  INFO BlockManagerMaster: Updated infoof  block 
broadcast_0_piece0
15/04/01  14:41:45  INFO BlockManagerInfo: Removed broadcast_0_piece0on  
192.168.1.5:63278  in  memory (size:869.0  B, free:530.0  MB)
15/04/01  14:41:45  INFO ContextCleaner: Cleaned broadcast0
15/04/01  14:41:46  INFO ObjectStore: Setting MetaStore object pin classeswith  
hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
15/04/01  14:41:46  INFO Datastore: The 
class"org.apache.hadoop.hive.metastore.model.MFieldSchema"  is  taggedas  
"embedded-only"  so doesnot  have its own datastoretable.
15/04/01  14:41:46  INFO Datastore: The 
class"org.apache.hadoop.hive.metastore.model.MOrder"  is  taggedas  
"embedded-only"  so doesnot  have its own datastoretable.
15/04/01  14:41:47  INFO Datastore: The 
class"org.apache.hadoop.hive.metastore.model.MFieldSchema"  is  taggedas  
"embedded-only"  so doesnot  have its own datastoretable.
15/04/01  14:41:47  INFO Datastore: The 
class"org.apache.hadoop.hive.metastore.model.MOrder"  is  taggedas  
"embedded-only"  so doesnot  have its own datastoretable.
15/04/01  14:41:47  INFO Query: Readingin  resultsfor  
query"org.datanucleus.store.rdbms.query.SQLQuery@0"  since theconnection  
usedis  closing
15/04/01  14:41:47  INFO ObjectStore: Initialized ObjectStore
15/04/01  14:41:47  INFO HiveMetaStore: Added admin rolein  metastore
15/04/01  14:41:47  INFO HiveMetaStore: Addedpublic  rolein  metastore
15/04/01  14:41:48  INFO HiveMetaStore:No  user  is  addedin  admin role, since 
configis  empty
15/04/01  14:41:48  INFO SessionState:No  Tezsession  requiredat  this point. 
hive.execution.engine=mr.
15/04/01  14:41:49  INFO ParseDriver: Parsing command:SELECT  path, name,value, 
v1.peValue, v1.peName
  FROM  metric
  lateralview  json_tuple(pathElements,'name','value') v1
as  peName, peValue
15/04/01  14:41:49  INFO ParseDriver: Parse Completed
Exception  in  thread"main"  java.lang.ClassNotFoundException: json_tuple
 at  java.net.URLClassLoader$1.run(URLClassLoader.java:372)
 at  java.net.URLClassLoader$1.run(URLClassLoader.java:361)
 at  java.security.AccessController.doPrivileged(Native Method)
 at  java.net.URLClassLoader.findClass(URLClassLoader.java:360)
 at  java.lang.ClassLoader.loadClass(ClassLoader.java:424)
 at  java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 at  
org.apache.spark.sql.hive.HiveFunctionWrapper.createFunction(Shim13.scala:141)
 at  
org.apache.spark.sql.hive.HiveGenericUdtf.function$lzycompute(hiveUdfs.scala:261)
 at  org.apache.spark.sql.hive.HiveGenericUdtf.function(hiveUdfs.scala:261)
 at  
org.apache.spark.sql.hive.HiveGenericUdtf.outputInspector$lzycompute(hiveUdfs.scala:267)
 at  
org.apache.spark.sql.hive.HiveGenericUdtf.outputInspector(hiveUdfs.scala:267)
 at  
org.apache.spark.sql.hive.HiveGenericUdtf.outputDataTypes$lzycompute(hiveUdfs.scala:272)
 at  
org

Re: Spark Sql - Missing Jar ? json_tuple NoClassDefFoundError

[Todd Nist]

Hi Deepujain,

I did include the jar file, I believe it is hive-exe.jar, through the
--jars option:

./bin/spark-shell --master spark://radtech.io:7077
--total-executor-cores 2 --driver-class-path
/usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar --jars
/opt/apache-hive-0.13.1-bin/lib/hive-exec-0.13.1.jar

Results in the same error.  I'm going to do the rebuild in a few minutes.

Thanks for the assistance.

-Todd



On Fri, Apr 3, 2015 at 6:30 AM, ÐΞ€ρ@Ҝ (๏̯͡๏)  wrote:

> I think you need to include the jar file through --jars option that
> contains the hive definition (code) of UDF json_tuple. That should solve
> your problem.
>
> On Fri, Apr 3, 2015 at 3:57 PM, Todd Nist  wrote:
>
>> I placed it there.  It was downloaded from MySql site.
>>
>> On Fri, Apr 3, 2015 at 6:25 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) 
>> wrote:
>>
>>> Akhil
>>> you mentioned /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar
>>> . how come you got this lib into spark/lib folder.
>>> 1) did you place it there ?
>>> 2) What is download location ?
>>>
>>>
>>> On Fri, Apr 3, 2015 at 3:42 PM, Todd Nist  wrote:
>>>
 Started the spark shell with the one jar from hive suggested:

 ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 
 2 --driver-class-path 
 /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar --jars 
 /opt/apache-hive-0.13.1-bin/lib/hive-exec-0.13.1.jar

 Results in the same error:

 scala> sql( | """SELECT path, name, value, v1.peValue, v1.peName   
   |  FROM metric_table |lateral view 
 json_tuple(pathElements, 'name', 'value') v1 |  as peName, 
 peValue | """)
 15/04/03 06:01:30 INFO ParseDriver: Parsing command: SELECT path, name, 
 value, v1.peValue, v1.peName FROM metric_table   lateral 
 view json_tuple(pathElements, 'name', 'value') v1 as peName, 
 peValue
 15/04/03 06:01:31 INFO ParseDriver: Parse Completed
 res2: org.apache.spark.sql.SchemaRDD =
 SchemaRDD[5] at RDD at SchemaRDD.scala:108== Query Plan  Physical Plan 
 ==
 java.lang.ClassNotFoundException: json_tuple

 I will try the rebuild.  Thanks again for the assistance.

 -Todd


 On Fri, Apr 3, 2015 at 5:34 AM, Akhil Das 
 wrote:

> Can you try building Spark
> 
> with hive support? Before that try to run the following:
>
> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-
> cores 2 --driver-class-path /usr/local/spark/lib/mysql-connector-java-
> 5.1.34-bin.jar --jars /opt/hive/0.13.1/lib/hive-exec.jar
>
> Thanks
> Best Regards
>
> On Fri, Apr 3, 2015 at 2:55 PM, Todd Nist  wrote:
>
>> Hi Akhil,
>>
>> This is for version 1.2.1.  Well the other thread that you reference
>> was me attempting it in 1.3.0 to see if the issue was related to 1.2.1.  
>> I
>> did not build Spark but used the version from the Spark download site for
>> 1.2.1 Pre Built for Hadoop 2.4 or Later.
>>
>> Since I get the error in both 1.2.1 and 1.3.0,
>>
>> 15/04/01 14:41:49 INFO ParseDriver: Parse Completed Exception in
>> thread "main" java.lang.ClassNotFoundException: json_tuple at
>> java.net.URLClassLoader$1.run(
>>
>> It looks like I just don't have the jar.  Even including all jars in
>> the $HIVE/lib directory did not seem to work.  Though when looking in
>> $HIVE/lib for 0.13.1, I do not see any json serde or jackson files.  I do
>> see that hive-exec.jar contains
>> the org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple class.  Do
>> you know if there is another Jar that is required or should it work just 
>> by
>> including all jars from $HIVE/lib?
>>
>> I can build it locally, but did not think that was required based on
>> the version I downloaded; is that not the case?
>>
>> Thanks for the assistance.
>>
>> -Todd
>>
>>
>> On Fri, Apr 3, 2015 at 2:06 AM, Akhil Das > > wrote:
>>
>>> How did you build spark? which version of spark are you having?
>>> Doesn't this thread already explains it?
>>> https://www.mail-archive.com/user@spark.apache.org/msg25505.html
>>>
>>> Thanks
>>> Best Regards
>>>
>>> On Thu, Apr 2, 2015 at 11:10 PM, Todd Nist 
>>> wrote:
>>>
 Hi Akhil,

 Tried your suggestion to no avail.  I actually to not see and
 "jackson" or "json serde" jars in the $HIVE/lib directory.  This is 
 hive
 0.13.1 and spark 1.2.1

 Here is what I did:

 I have added the lib folder to the –jars option when starting the
 spark-shell,
 but the job fails. The hive-site.xml is in the

Re: Spark Sql - Missing Jar ? json_tuple NoClassDefFoundError

[๏̯͡๏]

I think you need to include the jar file through --jars option that
contains the hive definition (code) of UDF json_tuple. That should solve
your problem.

On Fri, Apr 3, 2015 at 3:57 PM, Todd Nist  wrote:

> I placed it there.  It was downloaded from MySql site.
>
> On Fri, Apr 3, 2015 at 6:25 AM, ÐΞ€ρ@Ҝ (๏̯͡๏)  wrote:
>
>> Akhil
>> you mentioned /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar .
>> how come you got this lib into spark/lib folder.
>> 1) did you place it there ?
>> 2) What is download location ?
>>
>>
>> On Fri, Apr 3, 2015 at 3:42 PM, Todd Nist  wrote:
>>
>>> Started the spark shell with the one jar from hive suggested:
>>>
>>> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 2 
>>> --driver-class-path 
>>> /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar --jars 
>>> /opt/apache-hive-0.13.1-bin/lib/hive-exec-0.13.1.jar
>>>
>>> Results in the same error:
>>>
>>> scala> sql( | """SELECT path, name, value, v1.peValue, v1.peName
>>>  |  FROM metric_table |lateral view 
>>> json_tuple(pathElements, 'name', 'value') v1 |  as peName, 
>>> peValue | """)
>>> 15/04/03 06:01:30 INFO ParseDriver: Parsing command: SELECT path, name, 
>>> value, v1.peValue, v1.peName FROM metric_table   lateral 
>>> view json_tuple(pathElements, 'name', 'value') v1 as peName, 
>>> peValue
>>> 15/04/03 06:01:31 INFO ParseDriver: Parse Completed
>>> res2: org.apache.spark.sql.SchemaRDD =
>>> SchemaRDD[5] at RDD at SchemaRDD.scala:108== Query Plan  Physical Plan 
>>> ==
>>> java.lang.ClassNotFoundException: json_tuple
>>>
>>> I will try the rebuild.  Thanks again for the assistance.
>>>
>>> -Todd
>>>
>>>
>>> On Fri, Apr 3, 2015 at 5:34 AM, Akhil Das 
>>> wrote:
>>>
 Can you try building Spark
 
 with hive support? Before that try to run the following:

 ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-
 cores 2 --driver-class-path /usr/local/spark/lib/mysql-connector-java-5
 .1.34-bin.jar --jars /opt/hive/0.13.1/lib/hive-exec.jar

 Thanks
 Best Regards

 On Fri, Apr 3, 2015 at 2:55 PM, Todd Nist  wrote:

> Hi Akhil,
>
> This is for version 1.2.1.  Well the other thread that you reference
> was me attempting it in 1.3.0 to see if the issue was related to 1.2.1.  I
> did not build Spark but used the version from the Spark download site for
> 1.2.1 Pre Built for Hadoop 2.4 or Later.
>
> Since I get the error in both 1.2.1 and 1.3.0,
>
> 15/04/01 14:41:49 INFO ParseDriver: Parse Completed Exception in
> thread "main" java.lang.ClassNotFoundException: json_tuple at
> java.net.URLClassLoader$1.run(
>
> It looks like I just don't have the jar.  Even including all jars in
> the $HIVE/lib directory did not seem to work.  Though when looking in
> $HIVE/lib for 0.13.1, I do not see any json serde or jackson files.  I do
> see that hive-exec.jar contains
> the org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple class.  Do
> you know if there is another Jar that is required or should it work just 
> by
> including all jars from $HIVE/lib?
>
> I can build it locally, but did not think that was required based on
> the version I downloaded; is that not the case?
>
> Thanks for the assistance.
>
> -Todd
>
>
> On Fri, Apr 3, 2015 at 2:06 AM, Akhil Das 
> wrote:
>
>> How did you build spark? which version of spark are you having?
>> Doesn't this thread already explains it?
>> https://www.mail-archive.com/user@spark.apache.org/msg25505.html
>>
>> Thanks
>> Best Regards
>>
>> On Thu, Apr 2, 2015 at 11:10 PM, Todd Nist 
>> wrote:
>>
>>> Hi Akhil,
>>>
>>> Tried your suggestion to no avail.  I actually to not see and
>>> "jackson" or "json serde" jars in the $HIVE/lib directory.  This is hive
>>> 0.13.1 and spark 1.2.1
>>>
>>> Here is what I did:
>>>
>>> I have added the lib folder to the –jars option when starting the
>>> spark-shell,
>>> but the job fails. The hive-site.xml is in the $SPARK_HOME/conf
>>> directory.
>>>
>>> I start the spark-shell as follows:
>>>
>>> ./bin/spark-shell --master spark://radtech.io:7077 
>>> --total-executor-cores 2 --driver-class-path 
>>> /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar
>>>
>>> and like this
>>>
>>> ./bin/spark-shell --master spark://radtech.io:7077 
>>> --total-executor-cores 2 --driver-class-path 
>>> /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar --jars 
>>> /opt/hive/0.13.1/lib/*
>>>
>>> I’m just doing this in the spark-shell now:
>>>
>>> import org.apach

Re: Spark Sql - Missing Jar ? json_tuple NoClassDefFoundError

[Todd Nist]

I placed it there.  It was downloaded from MySql site.

On Fri, Apr 3, 2015 at 6:25 AM, ÐΞ€ρ@Ҝ (๏̯͡๏)  wrote:

> Akhil
> you mentioned /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar .
> how come you got this lib into spark/lib folder.
> 1) did you place it there ?
> 2) What is download location ?
>
>
> On Fri, Apr 3, 2015 at 3:42 PM, Todd Nist  wrote:
>
>> Started the spark shell with the one jar from hive suggested:
>>
>> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 2 
>> --driver-class-path /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar 
>> --jars /opt/apache-hive-0.13.1-bin/lib/hive-exec-0.13.1.jar
>>
>> Results in the same error:
>>
>> scala> sql( | """SELECT path, name, value, v1.peValue, v1.peName 
>> |  FROM metric_table |lateral view 
>> json_tuple(pathElements, 'name', 'value') v1 |  as peName, 
>> peValue | """)
>> 15/04/03 06:01:30 INFO ParseDriver: Parsing command: SELECT path, name, 
>> value, v1.peValue, v1.peName FROM metric_table   lateral 
>> view json_tuple(pathElements, 'name', 'value') v1 as peName, 
>> peValue
>> 15/04/03 06:01:31 INFO ParseDriver: Parse Completed
>> res2: org.apache.spark.sql.SchemaRDD =
>> SchemaRDD[5] at RDD at SchemaRDD.scala:108== Query Plan  Physical Plan ==
>> java.lang.ClassNotFoundException: json_tuple
>>
>> I will try the rebuild.  Thanks again for the assistance.
>>
>> -Todd
>>
>>
>> On Fri, Apr 3, 2015 at 5:34 AM, Akhil Das 
>> wrote:
>>
>>> Can you try building Spark
>>> 
>>> with hive support? Before that try to run the following:
>>>
>>> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-
>>> cores 2 --driver-class-path /usr/local/spark/lib/mysql-connector-java-5.
>>> 1.34-bin.jar --jars /opt/hive/0.13.1/lib/hive-exec.jar
>>>
>>> Thanks
>>> Best Regards
>>>
>>> On Fri, Apr 3, 2015 at 2:55 PM, Todd Nist  wrote:
>>>
 Hi Akhil,

 This is for version 1.2.1.  Well the other thread that you reference
 was me attempting it in 1.3.0 to see if the issue was related to 1.2.1.  I
 did not build Spark but used the version from the Spark download site for
 1.2.1 Pre Built for Hadoop 2.4 or Later.

 Since I get the error in both 1.2.1 and 1.3.0,

 15/04/01 14:41:49 INFO ParseDriver: Parse Completed Exception in
 thread "main" java.lang.ClassNotFoundException: json_tuple at
 java.net.URLClassLoader$1.run(

 It looks like I just don't have the jar.  Even including all jars in
 the $HIVE/lib directory did not seem to work.  Though when looking in
 $HIVE/lib for 0.13.1, I do not see any json serde or jackson files.  I do
 see that hive-exec.jar contains
 the org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple class.  Do
 you know if there is another Jar that is required or should it work just by
 including all jars from $HIVE/lib?

 I can build it locally, but did not think that was required based on
 the version I downloaded; is that not the case?

 Thanks for the assistance.

 -Todd


 On Fri, Apr 3, 2015 at 2:06 AM, Akhil Das 
 wrote:

> How did you build spark? which version of spark are you having?
> Doesn't this thread already explains it?
> https://www.mail-archive.com/user@spark.apache.org/msg25505.html
>
> Thanks
> Best Regards
>
> On Thu, Apr 2, 2015 at 11:10 PM, Todd Nist  wrote:
>
>> Hi Akhil,
>>
>> Tried your suggestion to no avail.  I actually to not see and
>> "jackson" or "json serde" jars in the $HIVE/lib directory.  This is hive
>> 0.13.1 and spark 1.2.1
>>
>> Here is what I did:
>>
>> I have added the lib folder to the –jars option when starting the
>> spark-shell,
>> but the job fails. The hive-site.xml is in the $SPARK_HOME/conf
>> directory.
>>
>> I start the spark-shell as follows:
>>
>> ./bin/spark-shell --master spark://radtech.io:7077 
>> --total-executor-cores 2 --driver-class-path 
>> /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar
>>
>> and like this
>>
>> ./bin/spark-shell --master spark://radtech.io:7077 
>> --total-executor-cores 2 --driver-class-path 
>> /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar --jars 
>> /opt/hive/0.13.1/lib/*
>>
>> I’m just doing this in the spark-shell now:
>>
>> import org.apache.spark.sql.hive._val sqlContext = new 
>> HiveContext(sc)import sqlContext._case class MetricTable(path: String, 
>> pathElements: String, name: String, value: String)val mt = new 
>> MetricTable("""path": "/DC1/HOST1/""",
>> """pathElements": [{"node": "DataCenter","value": "DC1"},{"node": 
>> "host","value": "HOST1"}]""",
>> 

Re: Spark Sql - Missing Jar ? json_tuple NoClassDefFoundError

[Akhil Das]

Copy pasted his command in the same thread.

Thanks
Best Regards

On Fri, Apr 3, 2015 at 3:55 PM, ÐΞ€ρ@Ҝ (๏̯͡๏)  wrote:

> Akhil
> you mentioned /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar .
> how come you got this lib into spark/lib folder.
> 1) did you place it there ?
> 2) What is download location ?
>
>
> On Fri, Apr 3, 2015 at 3:42 PM, Todd Nist  wrote:
>
>> Started the spark shell with the one jar from hive suggested:
>>
>> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 2 
>> --driver-class-path /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar 
>> --jars /opt/apache-hive-0.13.1-bin/lib/hive-exec-0.13.1.jar
>>
>> Results in the same error:
>>
>> scala> sql( | """SELECT path, name, value, v1.peValue, v1.peName 
>> |  FROM metric_table |lateral view 
>> json_tuple(pathElements, 'name', 'value') v1 |  as peName, 
>> peValue | """)
>> 15/04/03 06:01:30 INFO ParseDriver: Parsing command: SELECT path, name, 
>> value, v1.peValue, v1.peName FROM metric_table   lateral 
>> view json_tuple(pathElements, 'name', 'value') v1 as peName, 
>> peValue
>> 15/04/03 06:01:31 INFO ParseDriver: Parse Completed
>> res2: org.apache.spark.sql.SchemaRDD =
>> SchemaRDD[5] at RDD at SchemaRDD.scala:108== Query Plan  Physical Plan ==
>> java.lang.ClassNotFoundException: json_tuple
>>
>> I will try the rebuild.  Thanks again for the assistance.
>>
>> -Todd
>>
>>
>> On Fri, Apr 3, 2015 at 5:34 AM, Akhil Das 
>> wrote:
>>
>>> Can you try building Spark
>>> 
>>> with hive support? Before that try to run the following:
>>>
>>> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-
>>> cores 2 --driver-class-path /usr/local/spark/lib/mysql-connector-java-5.
>>> 1.34-bin.jar --jars /opt/hive/0.13.1/lib/hive-exec.jar
>>>
>>> Thanks
>>> Best Regards
>>>
>>> On Fri, Apr 3, 2015 at 2:55 PM, Todd Nist  wrote:
>>>
 Hi Akhil,

 This is for version 1.2.1.  Well the other thread that you reference
 was me attempting it in 1.3.0 to see if the issue was related to 1.2.1.  I
 did not build Spark but used the version from the Spark download site for
 1.2.1 Pre Built for Hadoop 2.4 or Later.

 Since I get the error in both 1.2.1 and 1.3.0,

 15/04/01 14:41:49 INFO ParseDriver: Parse Completed Exception in
 thread "main" java.lang.ClassNotFoundException: json_tuple at
 java.net.URLClassLoader$1.run(

 It looks like I just don't have the jar.  Even including all jars in
 the $HIVE/lib directory did not seem to work.  Though when looking in
 $HIVE/lib for 0.13.1, I do not see any json serde or jackson files.  I do
 see that hive-exec.jar contains
 the org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple class.  Do
 you know if there is another Jar that is required or should it work just by
 including all jars from $HIVE/lib?

 I can build it locally, but did not think that was required based on
 the version I downloaded; is that not the case?

 Thanks for the assistance.

 -Todd


 On Fri, Apr 3, 2015 at 2:06 AM, Akhil Das 
 wrote:

> How did you build spark? which version of spark are you having?
> Doesn't this thread already explains it?
> https://www.mail-archive.com/user@spark.apache.org/msg25505.html
>
> Thanks
> Best Regards
>
> On Thu, Apr 2, 2015 at 11:10 PM, Todd Nist  wrote:
>
>> Hi Akhil,
>>
>> Tried your suggestion to no avail.  I actually to not see and
>> "jackson" or "json serde" jars in the $HIVE/lib directory.  This is hive
>> 0.13.1 and spark 1.2.1
>>
>> Here is what I did:
>>
>> I have added the lib folder to the –jars option when starting the
>> spark-shell,
>> but the job fails. The hive-site.xml is in the $SPARK_HOME/conf
>> directory.
>>
>> I start the spark-shell as follows:
>>
>> ./bin/spark-shell --master spark://radtech.io:7077 
>> --total-executor-cores 2 --driver-class-path 
>> /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar
>>
>> and like this
>>
>> ./bin/spark-shell --master spark://radtech.io:7077 
>> --total-executor-cores 2 --driver-class-path 
>> /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar --jars 
>> /opt/hive/0.13.1/lib/*
>>
>> I’m just doing this in the spark-shell now:
>>
>> import org.apache.spark.sql.hive._val sqlContext = new 
>> HiveContext(sc)import sqlContext._case class MetricTable(path: String, 
>> pathElements: String, name: String, value: String)val mt = new 
>> MetricTable("""path": "/DC1/HOST1/""",
>> """pathElements": [{"node": "DataCenter","value": "DC1"},{"node": 
>> "host","value": "HOST1"}]""

Re: Spark Sql - Missing Jar ? json_tuple NoClassDefFoundError

[๏̯͡๏]

Akhil
you mentioned /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar .
how come you got this lib into spark/lib folder.
1) did you place it there ?
2) What is download location ?


On Fri, Apr 3, 2015 at 3:42 PM, Todd Nist  wrote:

> Started the spark shell with the one jar from hive suggested:
>
> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 2 
> --driver-class-path /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar 
> --jars /opt/apache-hive-0.13.1-bin/lib/hive-exec-0.13.1.jar
>
> Results in the same error:
>
> scala> sql( | """SELECT path, name, value, v1.peValue, v1.peName 
> |  FROM metric_table |lateral view 
> json_tuple(pathElements, 'name', 'value') v1 |  as peName, 
> peValue | """)
> 15/04/03 06:01:30 INFO ParseDriver: Parsing command: SELECT path, name, 
> value, v1.peValue, v1.peName FROM metric_table   lateral view 
> json_tuple(pathElements, 'name', 'value') v1 as peName, peValue
> 15/04/03 06:01:31 INFO ParseDriver: Parse Completed
> res2: org.apache.spark.sql.SchemaRDD =
> SchemaRDD[5] at RDD at SchemaRDD.scala:108== Query Plan  Physical Plan ==
> java.lang.ClassNotFoundException: json_tuple
>
> I will try the rebuild.  Thanks again for the assistance.
>
> -Todd
>
>
> On Fri, Apr 3, 2015 at 5:34 AM, Akhil Das 
> wrote:
>
>> Can you try building Spark
>> 
>> with hive support? Before that try to run the following:
>>
>> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores
>> 2 --driver-class-path /usr/local/spark/lib/mysql-connector-java-5.1.34-
>> bin.jar --jars /opt/hive/0.13.1/lib/hive-exec.jar
>>
>> Thanks
>> Best Regards
>>
>> On Fri, Apr 3, 2015 at 2:55 PM, Todd Nist  wrote:
>>
>>> Hi Akhil,
>>>
>>> This is for version 1.2.1.  Well the other thread that you reference was
>>> me attempting it in 1.3.0 to see if the issue was related to 1.2.1.  I did
>>> not build Spark but used the version from the Spark download site for 1.2.1
>>> Pre Built for Hadoop 2.4 or Later.
>>>
>>> Since I get the error in both 1.2.1 and 1.3.0,
>>>
>>> 15/04/01 14:41:49 INFO ParseDriver: Parse Completed Exception in thread
>>> "main" java.lang.ClassNotFoundException: json_tuple at
>>> java.net.URLClassLoader$1.run(
>>>
>>> It looks like I just don't have the jar.  Even including all jars in the
>>> $HIVE/lib directory did not seem to work.  Though when looking in $HIVE/lib
>>> for 0.13.1, I do not see any json serde or jackson files.  I do see that
>>> hive-exec.jar contains
>>> the org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple class.  Do
>>> you know if there is another Jar that is required or should it work just by
>>> including all jars from $HIVE/lib?
>>>
>>> I can build it locally, but did not think that was required based on the
>>> version I downloaded; is that not the case?
>>>
>>> Thanks for the assistance.
>>>
>>> -Todd
>>>
>>>
>>> On Fri, Apr 3, 2015 at 2:06 AM, Akhil Das 
>>> wrote:
>>>
 How did you build spark? which version of spark are you having? Doesn't
 this thread already explains it?
 https://www.mail-archive.com/user@spark.apache.org/msg25505.html

 Thanks
 Best Regards

 On Thu, Apr 2, 2015 at 11:10 PM, Todd Nist  wrote:

> Hi Akhil,
>
> Tried your suggestion to no avail.  I actually to not see and
> "jackson" or "json serde" jars in the $HIVE/lib directory.  This is hive
> 0.13.1 and spark 1.2.1
>
> Here is what I did:
>
> I have added the lib folder to the –jars option when starting the
> spark-shell,
> but the job fails. The hive-site.xml is in the $SPARK_HOME/conf
> directory.
>
> I start the spark-shell as follows:
>
> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 
> 2 --driver-class-path 
> /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar
>
> and like this
>
> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 
> 2 --driver-class-path 
> /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar --jars 
> /opt/hive/0.13.1/lib/*
>
> I’m just doing this in the spark-shell now:
>
> import org.apache.spark.sql.hive._val sqlContext = new 
> HiveContext(sc)import sqlContext._case class MetricTable(path: String, 
> pathElements: String, name: String, value: String)val mt = new 
> MetricTable("""path": "/DC1/HOST1/""",
> """pathElements": [{"node": "DataCenter","value": "DC1"},{"node": 
> "host","value": "HOST1"}]""",
> """name": "Memory Usage (%)""",
> """value": 29.590943279257175""")val rdd1 = sc.makeRDD(List(mt))
> rdd1.printSchema()
> rdd1.registerTempTable("metric_table")
> sql(
> """SELECT path, name, value, v1.peValue, v1.pe

Re: Spark Sql - Missing Jar ? json_tuple NoClassDefFoundError

[Todd Nist]

Started the spark shell with the one jar from hive suggested:

./bin/spark-shell --master spark://radtech.io:7077
--total-executor-cores 2 --driver-class-path
/usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar --jars
/opt/apache-hive-0.13.1-bin/lib/hive-exec-0.13.1.jar

Results in the same error:

scala> sql( | """SELECT path, name, value, v1.peValue,
v1.peName |  FROM metric_table |lateral
view json_tuple(pathElements, 'name', 'value') v1 |
as peName, peValue | """)
15/04/03 06:01:30 INFO ParseDriver: Parsing command: SELECT path,
name, value, v1.peValue, v1.peName FROM metric_table
lateral view json_tuple(pathElements, 'name', 'value') v1
as peName, peValue
15/04/03 06:01:31 INFO ParseDriver: Parse Completed
res2: org.apache.spark.sql.SchemaRDD =
SchemaRDD[5] at RDD at SchemaRDD.scala:108== Query Plan  Physical Plan ==
java.lang.ClassNotFoundException: json_tuple

I will try the rebuild.  Thanks again for the assistance.

-Todd


On Fri, Apr 3, 2015 at 5:34 AM, Akhil Das 
wrote:

> Can you try building Spark
> 
> with hive support? Before that try to run the following:
>
> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores
> 2 --driver-class-path /usr/local/spark/lib/mysql-connector-java-5.1.34-bin
> .jar --jars /opt/hive/0.13.1/lib/hive-exec.jar
>
> Thanks
> Best Regards
>
> On Fri, Apr 3, 2015 at 2:55 PM, Todd Nist  wrote:
>
>> Hi Akhil,
>>
>> This is for version 1.2.1.  Well the other thread that you reference was
>> me attempting it in 1.3.0 to see if the issue was related to 1.2.1.  I did
>> not build Spark but used the version from the Spark download site for 1.2.1
>> Pre Built for Hadoop 2.4 or Later.
>>
>> Since I get the error in both 1.2.1 and 1.3.0,
>>
>> 15/04/01 14:41:49 INFO ParseDriver: Parse Completed Exception in thread
>> "main" java.lang.ClassNotFoundException: json_tuple at
>> java.net.URLClassLoader$1.run(
>>
>> It looks like I just don't have the jar.  Even including all jars in the
>> $HIVE/lib directory did not seem to work.  Though when looking in $HIVE/lib
>> for 0.13.1, I do not see any json serde or jackson files.  I do see that
>> hive-exec.jar contains
>> the org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple class.  Do
>> you know if there is another Jar that is required or should it work just by
>> including all jars from $HIVE/lib?
>>
>> I can build it locally, but did not think that was required based on the
>> version I downloaded; is that not the case?
>>
>> Thanks for the assistance.
>>
>> -Todd
>>
>>
>> On Fri, Apr 3, 2015 at 2:06 AM, Akhil Das 
>> wrote:
>>
>>> How did you build spark? which version of spark are you having? Doesn't
>>> this thread already explains it?
>>> https://www.mail-archive.com/user@spark.apache.org/msg25505.html
>>>
>>> Thanks
>>> Best Regards
>>>
>>> On Thu, Apr 2, 2015 at 11:10 PM, Todd Nist  wrote:
>>>
 Hi Akhil,

 Tried your suggestion to no avail.  I actually to not see and "jackson"
 or "json serde" jars in the $HIVE/lib directory.  This is hive 0.13.1 and
 spark 1.2.1

 Here is what I did:

 I have added the lib folder to the –jars option when starting the
 spark-shell,
 but the job fails. The hive-site.xml is in the $SPARK_HOME/conf
 directory.

 I start the spark-shell as follows:

 ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 
 2 --driver-class-path 
 /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar

 and like this

 ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 
 2 --driver-class-path 
 /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar --jars 
 /opt/hive/0.13.1/lib/*

 I’m just doing this in the spark-shell now:

 import org.apache.spark.sql.hive._val sqlContext = new 
 HiveContext(sc)import sqlContext._case class MetricTable(path: String, 
 pathElements: String, name: String, value: String)val mt = new 
 MetricTable("""path": "/DC1/HOST1/""",
 """pathElements": [{"node": "DataCenter","value": "DC1"},{"node": 
 "host","value": "HOST1"}]""",
 """name": "Memory Usage (%)""",
 """value": 29.590943279257175""")val rdd1 = sc.makeRDD(List(mt))
 rdd1.printSchema()
 rdd1.registerTempTable("metric_table")
 sql(
 """SELECT path, name, value, v1.peValue, v1.peName
  FROM metric_table
lateral view json_tuple(pathElements, 'name', 'value') v1
  as peName, peValue
 """)
 .collect.foreach(println(_))

 It results in the same error:

 15/04/02 12:33:59 INFO ParseDriver: Parsing command: SELECT path, name, 
 value, v1.peValue, v1.peName FROM metric_table   lateral 
 view json_tuple(p

Re: Spark Sql - Missing Jar ? json_tuple NoClassDefFoundError

[Akhil Das]

Can you try building Spark

with hive support? Before that try to run the following:

./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 2
--driver-class-path /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar
--jars /opt/hive/0.13.1/lib/hive-exec.jar

Thanks
Best Regards

On Fri, Apr 3, 2015 at 2:55 PM, Todd Nist  wrote:

> Hi Akhil,
>
> This is for version 1.2.1.  Well the other thread that you reference was
> me attempting it in 1.3.0 to see if the issue was related to 1.2.1.  I did
> not build Spark but used the version from the Spark download site for 1.2.1
> Pre Built for Hadoop 2.4 or Later.
>
> Since I get the error in both 1.2.1 and 1.3.0,
>
> 15/04/01 14:41:49 INFO ParseDriver: Parse Completed Exception in thread
> "main" java.lang.ClassNotFoundException: json_tuple at
> java.net.URLClassLoader$1.run(
>
> It looks like I just don't have the jar.  Even including all jars in the
> $HIVE/lib directory did not seem to work.  Though when looking in $HIVE/lib
> for 0.13.1, I do not see any json serde or jackson files.  I do see that
> hive-exec.jar contains
> the org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple class.  Do
> you know if there is another Jar that is required or should it work just by
> including all jars from $HIVE/lib?
>
> I can build it locally, but did not think that was required based on the
> version I downloaded; is that not the case?
>
> Thanks for the assistance.
>
> -Todd
>
>
> On Fri, Apr 3, 2015 at 2:06 AM, Akhil Das 
> wrote:
>
>> How did you build spark? which version of spark are you having? Doesn't
>> this thread already explains it?
>> https://www.mail-archive.com/user@spark.apache.org/msg25505.html
>>
>> Thanks
>> Best Regards
>>
>> On Thu, Apr 2, 2015 at 11:10 PM, Todd Nist  wrote:
>>
>>> Hi Akhil,
>>>
>>> Tried your suggestion to no avail.  I actually to not see and "jackson"
>>> or "json serde" jars in the $HIVE/lib directory.  This is hive 0.13.1 and
>>> spark 1.2.1
>>>
>>> Here is what I did:
>>>
>>> I have added the lib folder to the –jars option when starting the
>>> spark-shell,
>>> but the job fails. The hive-site.xml is in the $SPARK_HOME/conf
>>> directory.
>>>
>>> I start the spark-shell as follows:
>>>
>>> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 2 
>>> --driver-class-path /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar
>>>
>>> and like this
>>>
>>> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 2 
>>> --driver-class-path 
>>> /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar --jars 
>>> /opt/hive/0.13.1/lib/*
>>>
>>> I’m just doing this in the spark-shell now:
>>>
>>> import org.apache.spark.sql.hive._val sqlContext = new 
>>> HiveContext(sc)import sqlContext._case class MetricTable(path: String, 
>>> pathElements: String, name: String, value: String)val mt = new 
>>> MetricTable("""path": "/DC1/HOST1/""",
>>> """pathElements": [{"node": "DataCenter","value": "DC1"},{"node": 
>>> "host","value": "HOST1"}]""",
>>> """name": "Memory Usage (%)""",
>>> """value": 29.590943279257175""")val rdd1 = sc.makeRDD(List(mt))
>>> rdd1.printSchema()
>>> rdd1.registerTempTable("metric_table")
>>> sql(
>>> """SELECT path, name, value, v1.peValue, v1.peName
>>>  FROM metric_table
>>>lateral view json_tuple(pathElements, 'name', 'value') v1
>>>  as peName, peValue
>>> """)
>>> .collect.foreach(println(_))
>>>
>>> It results in the same error:
>>>
>>> 15/04/02 12:33:59 INFO ParseDriver: Parsing command: SELECT path, name, 
>>> value, v1.peValue, v1.peName FROM metric_table   lateral 
>>> view json_tuple(pathElements, 'name', 'value') v1 as peName, 
>>> peValue
>>> 15/04/02 12:34:00 INFO ParseDriver: Parse Completed
>>> res2: org.apache.spark.sql.SchemaRDD =
>>> SchemaRDD[5] at RDD at SchemaRDD.scala:108== Query Plan  Physical Plan 
>>> ==
>>> java.lang.ClassNotFoundException: json_tuple
>>>
>>> Any other suggestions or am I doing something else wrong here?
>>>
>>> -Todd
>>>
>>>
>>>
>>> On Thu, Apr 2, 2015 at 2:00 AM, Akhil Das 
>>> wrote:
>>>
 Try adding all the jars in your $HIVE/lib directory. If you want the
 specific jar, you could look fr jackson or json serde in it.

 Thanks
 Best Regards

 On Thu, Apr 2, 2015 at 12:49 AM, Todd Nist  wrote:

> I have a feeling I’m missing a Jar that provides the support or could
> this may be related to
> https://issues.apache.org/jira/browse/SPARK-5792. If it is a Jar
> where would I find that ? I would have thought in the $HIVE/lib folder, 
> but
> not sure which jar contains it.
>
> Error:
>
> Create Metric Temporary Table for querying15/04/01 14:41:44 INFO 
> HiveMetaStore: 0: Opening raw store with implemenation 
> class:org.apa

Re: Spark Sql - Missing Jar ? json_tuple NoClassDefFoundError

[Todd Nist]

Hi Akhil,

This is for version 1.2.1.  Well the other thread that you reference was me
attempting it in 1.3.0 to see if the issue was related to 1.2.1.  I did not
build Spark but used the version from the Spark download site for 1.2.1 Pre
Built for Hadoop 2.4 or Later.

Since I get the error in both 1.2.1 and 1.3.0,

15/04/01 14:41:49 INFO ParseDriver: Parse Completed Exception in thread
"main" java.lang.ClassNotFoundException: json_tuple at
java.net.URLClassLoader$1.run(

It looks like I just don't have the jar.  Even including all jars in the
$HIVE/lib directory did not seem to work.  Though when looking in $HIVE/lib
for 0.13.1, I do not see any json serde or jackson files.  I do see that
hive-exec.jar contains
the org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple class.  Do
you know if there is another Jar that is required or should it work just by
including all jars from $HIVE/lib?

I can build it locally, but did not think that was required based on the
version I downloaded; is that not the case?

Thanks for the assistance.

-Todd


On Fri, Apr 3, 2015 at 2:06 AM, Akhil Das 
wrote:

> How did you build spark? which version of spark are you having? Doesn't
> this thread already explains it?
> https://www.mail-archive.com/user@spark.apache.org/msg25505.html
>
> Thanks
> Best Regards
>
> On Thu, Apr 2, 2015 at 11:10 PM, Todd Nist  wrote:
>
>> Hi Akhil,
>>
>> Tried your suggestion to no avail.  I actually to not see and "jackson"
>> or "json serde" jars in the $HIVE/lib directory.  This is hive 0.13.1 and
>> spark 1.2.1
>>
>> Here is what I did:
>>
>> I have added the lib folder to the –jars option when starting the
>> spark-shell,
>> but the job fails. The hive-site.xml is in the $SPARK_HOME/conf directory.
>>
>> I start the spark-shell as follows:
>>
>> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 2 
>> --driver-class-path /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar
>>
>> and like this
>>
>> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 2 
>> --driver-class-path /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar 
>> --jars /opt/hive/0.13.1/lib/*
>>
>> I’m just doing this in the spark-shell now:
>>
>> import org.apache.spark.sql.hive._val sqlContext = new HiveContext(sc)import 
>> sqlContext._case class MetricTable(path: String, pathElements: String, name: 
>> String, value: String)val mt = new MetricTable("""path": "/DC1/HOST1/""",
>> """pathElements": [{"node": "DataCenter","value": "DC1"},{"node": 
>> "host","value": "HOST1"}]""",
>> """name": "Memory Usage (%)""",
>> """value": 29.590943279257175""")val rdd1 = sc.makeRDD(List(mt))
>> rdd1.printSchema()
>> rdd1.registerTempTable("metric_table")
>> sql(
>> """SELECT path, name, value, v1.peValue, v1.peName
>>  FROM metric_table
>>lateral view json_tuple(pathElements, 'name', 'value') v1
>>  as peName, peValue
>> """)
>> .collect.foreach(println(_))
>>
>> It results in the same error:
>>
>> 15/04/02 12:33:59 INFO ParseDriver: Parsing command: SELECT path, name, 
>> value, v1.peValue, v1.peName FROM metric_table   lateral 
>> view json_tuple(pathElements, 'name', 'value') v1 as peName, 
>> peValue
>> 15/04/02 12:34:00 INFO ParseDriver: Parse Completed
>> res2: org.apache.spark.sql.SchemaRDD =
>> SchemaRDD[5] at RDD at SchemaRDD.scala:108== Query Plan  Physical Plan ==
>> java.lang.ClassNotFoundException: json_tuple
>>
>> Any other suggestions or am I doing something else wrong here?
>>
>> -Todd
>>
>>
>>
>> On Thu, Apr 2, 2015 at 2:00 AM, Akhil Das 
>> wrote:
>>
>>> Try adding all the jars in your $HIVE/lib directory. If you want the
>>> specific jar, you could look fr jackson or json serde in it.
>>>
>>> Thanks
>>> Best Regards
>>>
>>> On Thu, Apr 2, 2015 at 12:49 AM, Todd Nist  wrote:
>>>
 I have a feeling I’m missing a Jar that provides the support or could
 this may be related to https://issues.apache.org/jira/browse/SPARK-5792.
 If it is a Jar where would I find that ? I would have thought in the
 $HIVE/lib folder, but not sure which jar contains it.

 Error:

 Create Metric Temporary Table for querying15/04/01 14:41:44 INFO 
 HiveMetaStore: 0: Opening raw store with implemenation 
 class:org.apache.hadoop.hive.metastore.ObjectStore15/04/01 14:41:44 INFO 
 ObjectStore: ObjectStore, initialize called15/04/01 14:41:45 INFO 
 Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will 
 be ignored15/04/01 14:41:45 INFO Persistence: Property 
 datanucleus.cache.level2 unknown - will be ignored15/04/01 14:41:45 INFO 
 BlockManager: Removing broadcast 015/04/01 14:41:45 INFO BlockManager: 
 Removing block broadcast_015/04/01 14:41:45 INFO MemoryStore: Block 
 broadcast_0 of size 1272 dropped from memory (free 278018571)15/04/01 
 14:41:45 INFO BlockManager: Removing block broadcast_0_piece015/04/01 
>>

Re: Spark Sql - Missing Jar ? json_tuple NoClassDefFoundError

[Akhil Das]

How did you build spark? which version of spark are you having? Doesn't
this thread already explains it?
https://www.mail-archive.com/user@spark.apache.org/msg25505.html

Thanks
Best Regards

On Thu, Apr 2, 2015 at 11:10 PM, Todd Nist  wrote:

> Hi Akhil,
>
> Tried your suggestion to no avail.  I actually to not see and "jackson" or
> "json serde" jars in the $HIVE/lib directory.  This is hive 0.13.1 and
> spark 1.2.1
>
> Here is what I did:
>
> I have added the lib folder to the –jars option when starting the
> spark-shell,
> but the job fails. The hive-site.xml is in the $SPARK_HOME/conf directory.
>
> I start the spark-shell as follows:
>
> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 2 
> --driver-class-path /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar
>
> and like this
>
> ./bin/spark-shell --master spark://radtech.io:7077 --total-executor-cores 2 
> --driver-class-path /usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar 
> --jars /opt/hive/0.13.1/lib/*
>
> I’m just doing this in the spark-shell now:
>
> import org.apache.spark.sql.hive._val sqlContext = new HiveContext(sc)import 
> sqlContext._case class MetricTable(path: String, pathElements: String, name: 
> String, value: String)val mt = new MetricTable("""path": "/DC1/HOST1/""",
> """pathElements": [{"node": "DataCenter","value": "DC1"},{"node": 
> "host","value": "HOST1"}]""",
> """name": "Memory Usage (%)""",
> """value": 29.590943279257175""")val rdd1 = sc.makeRDD(List(mt))
> rdd1.printSchema()
> rdd1.registerTempTable("metric_table")
> sql(
> """SELECT path, name, value, v1.peValue, v1.peName
>  FROM metric_table
>lateral view json_tuple(pathElements, 'name', 'value') v1
>  as peName, peValue
> """)
> .collect.foreach(println(_))
>
> It results in the same error:
>
> 15/04/02 12:33:59 INFO ParseDriver: Parsing command: SELECT path, name, 
> value, v1.peValue, v1.peName FROM metric_table   lateral view 
> json_tuple(pathElements, 'name', 'value') v1 as peName, peValue
> 15/04/02 12:34:00 INFO ParseDriver: Parse Completed
> res2: org.apache.spark.sql.SchemaRDD =
> SchemaRDD[5] at RDD at SchemaRDD.scala:108== Query Plan  Physical Plan ==
> java.lang.ClassNotFoundException: json_tuple
>
> Any other suggestions or am I doing something else wrong here?
>
> -Todd
>
>
>
> On Thu, Apr 2, 2015 at 2:00 AM, Akhil Das 
> wrote:
>
>> Try adding all the jars in your $HIVE/lib directory. If you want the
>> specific jar, you could look fr jackson or json serde in it.
>>
>> Thanks
>> Best Regards
>>
>> On Thu, Apr 2, 2015 at 12:49 AM, Todd Nist  wrote:
>>
>>> I have a feeling I’m missing a Jar that provides the support or could
>>> this may be related to https://issues.apache.org/jira/browse/SPARK-5792.
>>> If it is a Jar where would I find that ? I would have thought in the
>>> $HIVE/lib folder, but not sure which jar contains it.
>>>
>>> Error:
>>>
>>> Create Metric Temporary Table for querying15/04/01 14:41:44 INFO 
>>> HiveMetaStore: 0: Opening raw store with implemenation 
>>> class:org.apache.hadoop.hive.metastore.ObjectStore15/04/01 14:41:44 INFO 
>>> ObjectStore: ObjectStore, initialize called15/04/01 14:41:45 INFO 
>>> Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will 
>>> be ignored15/04/01 14:41:45 INFO Persistence: Property 
>>> datanucleus.cache.level2 unknown - will be ignored15/04/01 14:41:45 INFO 
>>> BlockManager: Removing broadcast 015/04/01 14:41:45 INFO BlockManager: 
>>> Removing block broadcast_015/04/01 14:41:45 INFO MemoryStore: Block 
>>> broadcast_0 of size 1272 dropped from memory (free 278018571)15/04/01 
>>> 14:41:45 INFO BlockManager: Removing block broadcast_0_piece015/04/01 
>>> 14:41:45 INFO MemoryStore: Block broadcast_0_piece0 of size 869 dropped 
>>> from memory (free 278019440)15/04/01 14:41:45 INFO BlockManagerInfo: 
>>> Removed broadcast_0_piece0 on 192.168.1.5:63230 in memory (size: 869.0 B, 
>>> free: 265.1 MB)15/04/01 14:41:45 INFO BlockManagerMaster: Updated info of 
>>> block broadcast_0_piece015/04/01 14:41:45 INFO BlockManagerInfo: Removed 
>>> broadcast_0_piece0 on 192.168.1.5:63278 in memory (size: 869.0 B, free: 
>>> 530.0 MB)15/04/01 14:41:45 INFO ContextCleaner: Cleaned broadcast 015/04/01 
>>> 14:41:46 INFO ObjectStore: Setting MetaStore object pin classes with 
>>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"15/04/01
>>>  14:41:46 INFO Datastore: The class 
>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as 
>>> "embedded-only" so does not have its own datastore table.15/04/01 14:41:46 
>>> INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" 
>>> is tagged as "embedded-only" so does not have its own datastore 
>>> table.15/04/01 14:41:47 INFO Datastore: The class 
>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as 
>>> "embedde

Re: Spark Sql - Missing Jar ? json_tuple NoClassDefFoundError

[Todd Nist]

Hi Akhil,

Tried your suggestion to no avail.  I actually to not see and "jackson" or
"json serde" jars in the $HIVE/lib directory.  This is hive 0.13.1 and
spark 1.2.1

Here is what I did:

I have added the lib folder to the –jars option when starting the
spark-shell,
but the job fails. The hive-site.xml is in the $SPARK_HOME/conf directory.

I start the spark-shell as follows:

./bin/spark-shell --master spark://radtech.io:7077
--total-executor-cores 2 --driver-class-path
/usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar

and like this

./bin/spark-shell --master spark://radtech.io:7077
--total-executor-cores 2 --driver-class-path
/usr/local/spark/lib/mysql-connector-java-5.1.34-bin.jar --jars
/opt/hive/0.13.1/lib/*

I’m just doing this in the spark-shell now:

import org.apache.spark.sql.hive._val sqlContext = new
HiveContext(sc)import sqlContext._case class MetricTable(path: String,
pathElements: String, name: String, value: String)val mt = new
MetricTable("""path": "/DC1/HOST1/""",
"""pathElements": [{"node": "DataCenter","value": "DC1"},{"node":
"host","value": "HOST1"}]""",
"""name": "Memory Usage (%)""",
"""value": 29.590943279257175""")val rdd1 = sc.makeRDD(List(mt))
rdd1.printSchema()
rdd1.registerTempTable("metric_table")
sql(
"""SELECT path, name, value, v1.peValue, v1.peName
 FROM metric_table
   lateral view json_tuple(pathElements, 'name', 'value') v1
 as peName, peValue
""")
.collect.foreach(println(_))

It results in the same error:

15/04/02 12:33:59 INFO ParseDriver: Parsing command: SELECT path,
name, value, v1.peValue, v1.peName FROM metric_table
lateral view json_tuple(pathElements, 'name', 'value') v1
as peName, peValue
15/04/02 12:34:00 INFO ParseDriver: Parse Completed
res2: org.apache.spark.sql.SchemaRDD =
SchemaRDD[5] at RDD at SchemaRDD.scala:108== Query Plan  Physical Plan ==
java.lang.ClassNotFoundException: json_tuple

Any other suggestions or am I doing something else wrong here?

-Todd



On Thu, Apr 2, 2015 at 2:00 AM, Akhil Das 
wrote:

> Try adding all the jars in your $HIVE/lib directory. If you want the
> specific jar, you could look fr jackson or json serde in it.
>
> Thanks
> Best Regards
>
> On Thu, Apr 2, 2015 at 12:49 AM, Todd Nist  wrote:
>
>> I have a feeling I’m missing a Jar that provides the support or could
>> this may be related to https://issues.apache.org/jira/browse/SPARK-5792.
>> If it is a Jar where would I find that ? I would have thought in the
>> $HIVE/lib folder, but not sure which jar contains it.
>>
>> Error:
>>
>> Create Metric Temporary Table for querying15/04/01 14:41:44 INFO 
>> HiveMetaStore: 0: Opening raw store with implemenation 
>> class:org.apache.hadoop.hive.metastore.ObjectStore15/04/01 14:41:44 INFO 
>> ObjectStore: ObjectStore, initialize called15/04/01 14:41:45 INFO 
>> Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be 
>> ignored15/04/01 14:41:45 INFO Persistence: Property datanucleus.cache.level2 
>> unknown - will be ignored15/04/01 14:41:45 INFO BlockManager: Removing 
>> broadcast 015/04/01 14:41:45 INFO BlockManager: Removing block 
>> broadcast_015/04/01 14:41:45 INFO MemoryStore: Block broadcast_0 of size 
>> 1272 dropped from memory (free 278018571)15/04/01 14:41:45 INFO 
>> BlockManager: Removing block broadcast_0_piece015/04/01 14:41:45 INFO 
>> MemoryStore: Block broadcast_0_piece0 of size 869 dropped from memory (free 
>> 278019440)15/04/01 14:41:45 INFO BlockManagerInfo: Removed 
>> broadcast_0_piece0 on 192.168.1.5:63230 in memory (size: 869.0 B, free: 
>> 265.1 MB)15/04/01 14:41:45 INFO BlockManagerMaster: Updated info of block 
>> broadcast_0_piece015/04/01 14:41:45 INFO BlockManagerInfo: Removed 
>> broadcast_0_piece0 on 192.168.1.5:63278 in memory (size: 869.0 B, free: 
>> 530.0 MB)15/04/01 14:41:45 INFO ContextCleaner: Cleaned broadcast 015/04/01 
>> 14:41:46 INFO ObjectStore: Setting MetaStore object pin classes with 
>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"15/04/01
>>  14:41:46 INFO Datastore: The class 
>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as 
>> "embedded-only" so does not have its own datastore table.15/04/01 14:41:46 
>> INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is 
>> tagged as "embedded-only" so does not have its own datastore table.15/04/01 
>> 14:41:47 INFO Datastore: The class 
>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as 
>> "embedded-only" so does not have its own datastore table.15/04/01 14:41:47 
>> INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is 
>> tagged as "embedded-only" so does not have its own datastore table.15/04/01 
>> 14:41:47 INFO Query: Reading in results for query 
>> "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is 
>> closing15/04/01 14:41:47 INFO ObjectStore: Initialized O

Re: Spark Sql - Missing Jar ? json_tuple NoClassDefFoundError

[Akhil Das]

Try adding all the jars in your $HIVE/lib directory. If you want the
specific jar, you could look fr jackson or json serde in it.

Thanks
Best Regards

On Thu, Apr 2, 2015 at 12:49 AM, Todd Nist  wrote:

> I have a feeling I’m missing a Jar that provides the support or could this
> may be related to https://issues.apache.org/jira/browse/SPARK-5792. If it
> is a Jar where would I find that ? I would have thought in the $HIVE/lib
> folder, but not sure which jar contains it.
>
> Error:
>
> Create Metric Temporary Table for querying15/04/01 14:41:44 INFO 
> HiveMetaStore: 0: Opening raw store with implemenation 
> class:org.apache.hadoop.hive.metastore.ObjectStore15/04/01 14:41:44 INFO 
> ObjectStore: ObjectStore, initialize called15/04/01 14:41:45 INFO 
> Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be 
> ignored15/04/01 14:41:45 INFO Persistence: Property datanucleus.cache.level2 
> unknown - will be ignored15/04/01 14:41:45 INFO BlockManager: Removing 
> broadcast 015/04/01 14:41:45 INFO BlockManager: Removing block 
> broadcast_015/04/01 14:41:45 INFO MemoryStore: Block broadcast_0 of size 1272 
> dropped from memory (free 278018571)15/04/01 14:41:45 INFO BlockManager: 
> Removing block broadcast_0_piece015/04/01 14:41:45 INFO MemoryStore: Block 
> broadcast_0_piece0 of size 869 dropped from memory (free 278019440)15/04/01 
> 14:41:45 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 
> 192.168.1.5:63230 in memory (size: 869.0 B, free: 265.1 MB)15/04/01 14:41:45 
> INFO BlockManagerMaster: Updated info of block broadcast_0_piece015/04/01 
> 14:41:45 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 
> 192.168.1.5:63278 in memory (size: 869.0 B, free: 530.0 MB)15/04/01 14:41:45 
> INFO ContextCleaner: Cleaned broadcast 015/04/01 14:41:46 INFO ObjectStore: 
> Setting MetaStore object pin classes with 
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"15/04/01
>  14:41:46 INFO Datastore: The class 
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as 
> "embedded-only" so does not have its own datastore table.15/04/01 14:41:46 
> INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is 
> tagged as "embedded-only" so does not have its own datastore table.15/04/01 
> 14:41:47 INFO Datastore: The class 
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as 
> "embedded-only" so does not have its own datastore table.15/04/01 14:41:47 
> INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is 
> tagged as "embedded-only" so does not have its own datastore table.15/04/01 
> 14:41:47 INFO Query: Reading in results for query 
> "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is 
> closing15/04/01 14:41:47 INFO ObjectStore: Initialized ObjectStore15/04/01 
> 14:41:47 INFO HiveMetaStore: Added admin role in metastore15/04/01 14:41:47 
> INFO HiveMetaStore: Added public role in metastore15/04/01 14:41:48 INFO 
> HiveMetaStore: No user is added in admin role, since config is empty15/04/01 
> 14:41:48 INFO SessionState: No Tez session required at this point. 
> hive.execution.engine=mr.15/04/01 14:41:49 INFO ParseDriver: Parsing command: 
> SELECT path, name, value, v1.peValue, v1.peName
>  FROM metric
>  lateral view json_tuple(pathElements, 'name', 'value') v1
>as peName, peValue15/04/01 14:41:49 INFO ParseDriver: Parse 
> CompletedException in thread "main" java.lang.ClassNotFoundException: 
> json_tuple
> at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at 
> org.apache.spark.sql.hive.HiveFunctionWrapper.createFunction(Shim13.scala:141)
> at 
> org.apache.spark.sql.hive.HiveGenericUdtf.function$lzycompute(hiveUdfs.scala:261)
> at org.apache.spark.sql.hive.HiveGenericUdtf.function(hiveUdfs.scala:261)
> at 
> org.apache.spark.sql.hive.HiveGenericUdtf.outputInspector$lzycompute(hiveUdfs.scala:267)
> at 
> org.apache.spark.sql.hive.HiveGenericUdtf.outputInspector(hiveUdfs.scala:267)
> at 
> org.apache.spark.sql.hive.HiveGenericUdtf.outputDataTypes$lzycompute(hiveUdfs.scala:272)
> at 
> org.apache.spark.sql.hive.HiveGenericUdtf.outputDataTypes(hiveUdfs.scala:272)
> at 
> org.apache.spark.sql.hive.HiveGenericUdtf.makeOutput(hiveUdfs.scala:278)
> at 
> org.apache.spark.sql.catalyst.expressions.Generator.output(generators.scala:60)
> at 
> org.apache.spark.sql.catalyst.plans.logical.Generate$$anonfun$1.apply(basicOperators.scala:50)
>   at 
> org.apache.spark.sql.catalyst.plans.logical.Generate$$anonfun$1.apply(basic

Spark Sql - Missing Jar ? json_tuple NoClassDefFoundError

[Todd Nist]

I have a feeling I’m missing a Jar that provides the support or could this
may be related to https://issues.apache.org/jira/browse/SPARK-5792. If it
is a Jar where would I find that ? I would have thought in the $HIVE/lib
folder, but not sure which jar contains it.

Error:

Create Metric Temporary Table for querying15/04/01 14:41:44 INFO
HiveMetaStore: 0: Opening raw store with implemenation
class:org.apache.hadoop.hive.metastore.ObjectStore15/04/01 14:41:44
INFO ObjectStore: ObjectStore, initialize called15/04/01 14:41:45 INFO
Persistence: Property hive.metastore.integral.jdo.pushdown unknown -
will be ignored15/04/01 14:41:45 INFO Persistence: Property
datanucleus.cache.level2 unknown - will be ignored15/04/01 14:41:45
INFO BlockManager: Removing broadcast 015/04/01 14:41:45 INFO
BlockManager: Removing block broadcast_015/04/01 14:41:45 INFO
MemoryStore: Block broadcast_0 of size 1272 dropped from memory (free
278018571)15/04/01 14:41:45 INFO BlockManager: Removing block
broadcast_0_piece015/04/01 14:41:45 INFO MemoryStore: Block
broadcast_0_piece0 of size 869 dropped from memory (free
278019440)15/04/01 14:41:45 INFO BlockManagerInfo: Removed
broadcast_0_piece0 on 192.168.1.5:63230 in memory (size: 869.0 B,
free: 265.1 MB)15/04/01 14:41:45 INFO BlockManagerMaster: Updated info
of block broadcast_0_piece015/04/01 14:41:45 INFO BlockManagerInfo:
Removed broadcast_0_piece0 on 192.168.1.5:63278 in memory (size: 869.0
B, free: 530.0 MB)15/04/01 14:41:45 INFO ContextCleaner: Cleaned
broadcast 015/04/01 14:41:46 INFO ObjectStore: Setting MetaStore
object pin classes with
hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"15/04/01
14:41:46 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.15/04/01
14:41:46 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
"embedded-only" so does not have its own datastore table.15/04/01
14:41:47 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.15/04/01
14:41:47 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
"embedded-only" so does not have its own datastore table.15/04/01
14:41:47 INFO Query: Reading in results for query
"org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection
used is closing15/04/01 14:41:47 INFO ObjectStore: Initialized
ObjectStore15/04/01 14:41:47 INFO HiveMetaStore: Added admin role in
metastore15/04/01 14:41:47 INFO HiveMetaStore: Added public role in
metastore15/04/01 14:41:48 INFO HiveMetaStore: No user is added in
admin role, since config is empty15/04/01 14:41:48 INFO SessionState:
No Tez session required at this point.
hive.execution.engine=mr.15/04/01 14:41:49 INFO ParseDriver: Parsing
command: SELECT path, name, value, v1.peValue, v1.peName
 FROM metric
 lateral view json_tuple(pathElements, 'name', 'value') v1
   as peName, peValue15/04/01 14:41:49 INFO ParseDriver:
Parse CompletedException in thread "main"
java.lang.ClassNotFoundException: json_tuple
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at 
org.apache.spark.sql.hive.HiveFunctionWrapper.createFunction(Shim13.scala:141)
at 
org.apache.spark.sql.hive.HiveGenericUdtf.function$lzycompute(hiveUdfs.scala:261)
at org.apache.spark.sql.hive.HiveGenericUdtf.function(hiveUdfs.scala:261)
at 
org.apache.spark.sql.hive.HiveGenericUdtf.outputInspector$lzycompute(hiveUdfs.scala:267)
at 
org.apache.spark.sql.hive.HiveGenericUdtf.outputInspector(hiveUdfs.scala:267)
at 
org.apache.spark.sql.hive.HiveGenericUdtf.outputDataTypes$lzycompute(hiveUdfs.scala:272)
at 
org.apache.spark.sql.hive.HiveGenericUdtf.outputDataTypes(hiveUdfs.scala:272)
at org.apache.spark.sql.hive.HiveGenericUdtf.makeOutput(hiveUdfs.scala:278)
at 
org.apache.spark.sql.catalyst.expressions.Generator.output(generators.scala:60)
at 
org.apache.spark.sql.catalyst.plans.logical.Generate$$anonfun$1.apply(basicOperators.scala:50)
at 
org.apache.spark.sql.catalyst.plans.logical.Generate$$anonfun$1.apply(basicOperators.scala:50)
at scala.Option.map(Option.scala:145)
at 
org.apache.spark.sql.catalyst.plans.logical.Generate.generatorOutput(basicOperators.scala:50)
at 
org.apache.spark.sql.catalyst.plans.logical.Generate.output(basicOperators.scala:60)
at 
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveChildren$1.apply(LogicalPlan.scala:118)
at 
org.apac