[
https://issues.apache.org/jira/browse/GRIFFIN-330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Obaidul Karim updated GRIFFIN-330:
----------------------------------
Description:
*Issue:*
Built Griffin 0.5.0 without UI & Unit tests. [mvn --projects measure
--also-make clean install -Dmaven.test.skip=true]
It failed to run sample code (similar to quick start code) on EMR 6.0.0 but the
same code was successful on EMR 5.30.1.
*Questions:*
1. Is it either issue with hadoop/hive version of scala version ?
2. If versions are the issue in that what is the plan for future support?
+*Failed Case:*+
Using below environment.
{code:java}
[EMR 6.0.0]
Hadoop 3.2.1
Hive 3.1.2 (aws glue metastore)
Spark 2.4.4
Scala version 2.12.10
Apache Maven 3.5.2
Java version: 1.8.0_242, vendor: Amazon.com Inc
OS name: "linux", version: "4.14.165-133.209.amzn2.x86_64", arch: "amd64",
family: "unix"
griffin-0.5.0 (mvn --projects measure --also-make clean install
-Dmaven.test.skip=true){code}
*Command:*
{code:java}
spark-submit --class org.apache.griffin.measure.Application --master yarn
--deploy-mode client \
--queue default --driver-memory 1g --executor-memory 1g --num-executors 2 \
/home/hadoop/griffin/griffin-0.5.0/measure/target/measure-0.5.0.jar
/home/hadoop/env.json /home/hadoop/dq.json{code}
*Error:*
{code:java}
20/06/30 05:56:51 INFO Application$: [Ljava.lang.String;@3561c410
20/06/30 05:56:51 INFO Application$: /home/hadoop/env.json
20/06/30 05:56:51 INFO Application$: /home/hadoop/dq.json
Exception in thread "main" java.lang.NoClassDefFoundError: scala/Product$class
at
org.apache.griffin.measure.configuration.dqdefinition.reader.ParamFileReader.<init>(ParamFileReader.scala:36)
at
org.apache.griffin.measure.configuration.dqdefinition.reader.ParamReaderFactory$.getParamReader(ParamReaderFactory.scala:36)
at org.apache.griffin.measure.Application$.readParamFile(Application.scala:122)
at org.apache.griffin.measure.Application$.main(Application.scala:51)
at org.apache.griffin.measure.Application.main(Application.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:853)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:928)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:937)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: scala.Product$class
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 17 more
20/06/30 05:56:51 INFO ShutdownHookManager: Shutdown hook called
20/06/30 05:56:51 INFO ShutdownHookManager: Deleting directory
/mnt/tmp/spark-20c5087b-8d2b-4358-92aa-7a398f7078ef{code}
**
+*Success Case:*+
However it is successful using below environment (older EMR):
{code:java}
[EMR 5.30.1]
Hadoop 2.8.5
Hive 2.3.6 (aws glue metastore)
Spark 2.4.5
Scala version 2.11.12
Apache Maven 3.5.2
Java version: 1.8.0_252, vendor: Amazon.com Inc.
OS name: "linux", version: "4.14.173-137.229.amzn2.x86_64", arch: "amd64",
family: "unix"
griffin-0.5.0 (mvn --projects measure --also-make clean install
-Dmaven.test.skip=true){code}
*Command:*
{code:java}
spark-submit --class org.apache.griffin.measure.Application --master yarn
--deploy-mode client \
--queue default --driver-memory 1g --executor-memory 1g --num-executors 2 \
/home/hadoop/griffin/griffin-0.5.0/measure/target/measure-0.5.0.jar
/home/hadoop/env.json /home/hadoop/dq.json{code}
*Output:*
{code:java}
data source timeRanges: src -> (1593499903348, 1593499903348], tgt ->
(1593499903348, 1593499903348]
[1593499903348] batch_accu start: application_1593497650758_0002
batch_accu [1593499903348] metrics:
{"name":"batch_accu","tmst":1593499903348,"value":{"total_count":10,"miss_count":3,"matched_count":7,"matchedFraction":0.7},"applicationId":"applicati
on_1593497650758_0002"}
[1593499903348] 1593499990953: process using time: 87605 ms
[1593499903348] batch_accu finish{code}
was:
*Issue:*
Built Griffin 0.5.0 without UI & Unit tests. It failed to run sample code
(similar to quick start code) on EMR 6.0.0 but the same code was successful on
EMR 5.30.1.
*Question2:*
1. Is it either issue with hadoop/hive version of scala version ?
2. If versions are the issue in that what is the plan for future support?
+*Failed Case:*+
Using below environment.
{code:java}
[EMR 6.0.0]
Hadoop 3.2.1
Hive 3.1.2 (aws glue metastore)
Spark 2.4.4
Scala version 2.12.10
Apache Maven 3.5.2
Java version: 1.8.0_242, vendor: Amazon.com Inc
OS name: "linux", version: "4.14.165-133.209.amzn2.x86_64", arch: "amd64",
family: "unix"
griffin-0.5.0 (mvn --projects measure --also-make clean install
-Dmaven.test.skip=true){code}
*Command:*
{code:java}
spark-submit --class org.apache.griffin.measure.Application --master yarn
--deploy-mode client \
--queue default --driver-memory 1g --executor-memory 1g --num-executors 2 \
/home/hadoop/griffin/griffin-0.5.0/measure/target/measure-0.5.0.jar
/home/hadoop/env.json /home/hadoop/dq.json{code}
**
*Error:*
{code:java}
20/06/30 05:56:51 INFO Application$: [Ljava.lang.String;@3561c410
20/06/30 05:56:51 INFO Application$: /home/hadoop/env.json
20/06/30 05:56:51 INFO Application$: /home/hadoop/dq.json
Exception in thread "main" java.lang.NoClassDefFoundError: scala/Product$class
at
org.apache.griffin.measure.configuration.dqdefinition.reader.ParamFileReader.<init>(ParamFileReader.scala:36)
at
org.apache.griffin.measure.configuration.dqdefinition.reader.ParamReaderFactory$.getParamReader(ParamReaderFactory.scala:36)
at org.apache.griffin.measure.Application$.readParamFile(Application.scala:122)
at org.apache.griffin.measure.Application$.main(Application.scala:51)
at org.apache.griffin.measure.Application.main(Application.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:853)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:928)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:937)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: scala.Product$class
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 17 more
20/06/30 05:56:51 INFO ShutdownHookManager: Shutdown hook called
20/06/30 05:56:51 INFO ShutdownHookManager: Deleting directory
/mnt/tmp/spark-20c5087b-8d2b-4358-92aa-7a398f7078ef{code}
**
+*Success Case:*+
However it is successful using below environment (older EMR):
{code:java}
[EMR 5.30.1]
Hadoop 2.8.5
Hive 2.3.6 (aws glue metastore)
Spark 2.4.5
Scala version 2.11.12
Apache Maven 3.5.2
Java version: 1.8.0_252, vendor: Amazon.com Inc.
OS name: "linux", version: "4.14.173-137.229.amzn2.x86_64", arch: "amd64",
family: "unix"
griffin-0.5.0 (mvn --projects measure --also-make clean install
-Dmaven.test.skip=true){code}
*Command:*
{code:java}
spark-submit --class org.apache.griffin.measure.Application --master yarn
--deploy-mode client \
--queue default --driver-memory 1g --executor-memory 1g --num-executors 2 \
/home/hadoop/griffin/griffin-0.5.0/measure/target/measure-0.5.0.jar
/home/hadoop/env.json /home/hadoop/dq.json{code}
*Output:*
{code:java}
data source timeRanges: src -> (1593499903348, 1593499903348], tgt ->
(1593499903348, 1593499903348]
[1593499903348] batch_accu start: application_1593497650758_0002
batch_accu [1593499903348] metrics:
{"name":"batch_accu","tmst":1593499903348,"value":{"total_count":10,"miss_count":3,"matched_count":7,"matchedFraction":0.7},"applicationId":"applicati
on_1593497650758_0002"}
[1593499903348] 1593499990953: process using time: 87605 ms
[1593499903348] batch_accu finish{code}
> Failed to Run on (EMR 6.0.0) - java.lang.NoClassDefFoundError:
> scala/Product$class
> ----------------------------------------------------------------------------------
>
> Key: GRIFFIN-330
> URL: https://issues.apache.org/jira/browse/GRIFFIN-330
> Project: Griffin
> Issue Type: Bug
> Affects Versions: 0.5.0
> Environment: Failed Environment:
> -------------------
> [EMR 6.0.0]
> Hadoop 3.2.1
> Hive 3.1.2 (aws glue metastore)
> Spark 2.4.4
> Scala version 2.12.10
> Apache Maven 3.5.2
> Java version: 1.8.0_242, vendor: Amazon.com Inc
> OS name: "linux", version: "4.14.165-133.209.amzn2.x86_64", arch: "amd64",
> family: "unix"
> griffin-0.5.0 (mvn --projects measure --also-make clean install
> -Dmaven.test.skip=true)
> Success Environment:
> ---------------------
> [EMR 5.30.1]
> Hadoop 2.8.5
> Hive 2.3.6 (aws glue metastore)
> Spark 2.4.5
> Scala version 2.11.12
> Apache Maven 3.5.2
> Java version: 1.8.0_252, vendor: Amazon.com Inc.
> OS name: "linux", version: "4.14.173-137.229.amzn2.x86_64", arch: "amd64",
> family: "unix"
> griffin-0.5.0 (mvn --projects measure --also-make clean install
> -Dmaven.test.skip=true)
> Reporter: Obaidul Karim
> Priority: Minor
> Labels: 0.5.0, 5.30.1, 6.0.0, EMR, hadoop3, hive3
>
> *Issue:*
> Built Griffin 0.5.0 without UI & Unit tests. [mvn --projects measure
> --also-make clean install -Dmaven.test.skip=true]
> It failed to run sample code (similar to quick start code) on EMR 6.0.0 but
> the same code was successful on EMR 5.30.1.
> *Questions:*
> 1. Is it either issue with hadoop/hive version of scala version ?
> 2. If versions are the issue in that what is the plan for future support?
> +*Failed Case:*+
> Using below environment.
> {code:java}
> [EMR 6.0.0]
> Hadoop 3.2.1
> Hive 3.1.2 (aws glue metastore)
> Spark 2.4.4
> Scala version 2.12.10
> Apache Maven 3.5.2
> Java version: 1.8.0_242, vendor: Amazon.com Inc
> OS name: "linux", version: "4.14.165-133.209.amzn2.x86_64", arch: "amd64",
> family: "unix"
> griffin-0.5.0 (mvn --projects measure --also-make clean install
> -Dmaven.test.skip=true){code}
> *Command:*
> {code:java}
> spark-submit --class org.apache.griffin.measure.Application --master yarn
> --deploy-mode client \
> --queue default --driver-memory 1g --executor-memory 1g --num-executors 2 \
> /home/hadoop/griffin/griffin-0.5.0/measure/target/measure-0.5.0.jar
> /home/hadoop/env.json /home/hadoop/dq.json{code}
> *Error:*
> {code:java}
> 20/06/30 05:56:51 INFO Application$: [Ljava.lang.String;@3561c410
> 20/06/30 05:56:51 INFO Application$: /home/hadoop/env.json
> 20/06/30 05:56:51 INFO Application$: /home/hadoop/dq.json
> Exception in thread "main" java.lang.NoClassDefFoundError: scala/Product$class
> at
> org.apache.griffin.measure.configuration.dqdefinition.reader.ParamFileReader.<init>(ParamFileReader.scala:36)
> at
> org.apache.griffin.measure.configuration.dqdefinition.reader.ParamReaderFactory$.getParamReader(ParamReaderFactory.scala:36)
> at
> org.apache.griffin.measure.Application$.readParamFile(Application.scala:122)
> at org.apache.griffin.measure.Application$.main(Application.scala:51)
> at org.apache.griffin.measure.Application.main(Application.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:853)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:928)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:937)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException: scala.Product$class
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
> ... 17 more
> 20/06/30 05:56:51 INFO ShutdownHookManager: Shutdown hook called
> 20/06/30 05:56:51 INFO ShutdownHookManager: Deleting directory
> /mnt/tmp/spark-20c5087b-8d2b-4358-92aa-7a398f7078ef{code}
> **
> +*Success Case:*+
> However it is successful using below environment (older EMR):
> {code:java}
> [EMR 5.30.1]
> Hadoop 2.8.5
> Hive 2.3.6 (aws glue metastore)
> Spark 2.4.5
> Scala version 2.11.12
> Apache Maven 3.5.2
> Java version: 1.8.0_252, vendor: Amazon.com Inc.
> OS name: "linux", version: "4.14.173-137.229.amzn2.x86_64", arch: "amd64",
> family: "unix"
> griffin-0.5.0 (mvn --projects measure --also-make clean install
> -Dmaven.test.skip=true){code}
> *Command:*
> {code:java}
> spark-submit --class org.apache.griffin.measure.Application --master yarn
> --deploy-mode client \
> --queue default --driver-memory 1g --executor-memory 1g --num-executors 2 \
> /home/hadoop/griffin/griffin-0.5.0/measure/target/measure-0.5.0.jar
> /home/hadoop/env.json /home/hadoop/dq.json{code}
> *Output:*
> {code:java}
> data source timeRanges: src -> (1593499903348, 1593499903348], tgt ->
> (1593499903348, 1593499903348]
> [1593499903348] batch_accu start: application_1593497650758_0002
> batch_accu [1593499903348] metrics:
> {"name":"batch_accu","tmst":1593499903348,"value":{"total_count":10,"miss_count":3,"matched_count":7,"matchedFraction":0.7},"applicationId":"applicati
> on_1593497650758_0002"}
> [1593499903348] 1593499990953: process using time: 87605 ms
> [1593499903348] batch_accu finish{code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
