[jira] [Comment Edited] (SPARK-1698) Improve spark integration
[
https://issues.apache.org/jira/browse/SPARK-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987699#comment-13987699
]
Guoqiang Li edited comment on SPARK-1698 at 5/2/14 2:32 PM:
[~srowen]
In [The PR 598|https://github.com/apache/spark/pull/598] #1,#2,#4 do not occur
and #3 is very easy to find.
The directory structure of spark similar to hadoop 2.3.0.
There are three subcomponents: core,examples,hive,The directory structure of
spark:
{code}
+- SPARK_HOME
| +- bin
| \- *
| +- sbin
| \- *
| +- RELEASE
| +- conf
| \- *
| +- python
| \- *
| +- share
| \- spark
|+- core
| +- lib
| \- *.jar
| +- spark-core*.jar
| +- spark-repl*.jar
| +- spark-yarn*.jar
| +- spark-bagel*.jar
| +- spark-graphx*.jar
| +- spark-sql*.jar
| +- spark-catalyst*.jar
| +- spark-mllib*.jar
| +- spark-streaming*.jar
|+- hive
| +- lib
| \- *.jar
| \- spark-hive*.jar
|+- examples
| +- lib
| \- *.jar
| \- spark-examples*.jar
{code}
was (Author: gq):
[~srowen]
In [The PR 598|https://github.com/apache/spark/pull/598] #1,#2,#4 do not occur
and #3 is very easy to find.
The directory structure of Spark similar to hadoop 2.3.0.
There are three subcomponents: core,examples,hive,their path is
share/spark/core,share/spark/examples,share/spark/hive. their dependency path
is share/spark/core/lib,share/spark/examples/lib,share/spark/hive/lib
Improve spark integration
-
Key: SPARK-1698
URL: https://issues.apache.org/jira/browse/SPARK-1698
Project: Spark
Issue Type: Improvement
Components: Build, Deploy
Reporter: Guoqiang Li
Assignee: Guoqiang Li
Fix For: 1.0.0
Use the shade plugin to create a big JAR with all the dependencies can cause
a few problems
1. Missing jar's meta information
2. Some file is covered, eg: plugin.xml
3. Different versions of a jar may co-exist
4. Too big, java 6 does not support
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (SPARK-1698) Improve spark integration
[
https://issues.apache.org/jira/browse/SPARK-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987699#comment-13987699
]
Guoqiang Li edited comment on SPARK-1698 at 5/2/14 2:36 PM:
[~srowen]
In [The PR 598|https://github.com/apache/spark/pull/598] #1,#2,#4 do not occur
and #3 is very easy to find.
The directory structure of spark similar to hadoop 2.3.0.
There are three subcomponents: core,examples,hive,The directory structure of
spark:
{code}
+- SPARK_HOME
| +- bin
| \- *
| +- sbin
| \- *
| +- RELEASE
| +- conf
| \- *
| +- python
| \- *
| \- share
| \- spark
|+- core
| +- lib
| \- *.jar
| +- spark-core*.jar
| +- spark-repl*.jar
| +- spark-yarn*.jar
| +- spark-bagel*.jar
| +- spark-graphx*.jar
| +- spark-sql*.jar
| +- spark-catalyst*.jar
| +- spark-mllib*.jar
| \- spark-streaming*.jar
|+- hive
| +- lib
| \- *.jar
| \- spark-hive*.jar
|+- examples
| +- lib
| \- *.jar
| \- spark-examples*.jar
{code}
was (Author: gq):
[~srowen]
In [The PR 598|https://github.com/apache/spark/pull/598] #1,#2,#4 do not occur
and #3 is very easy to find.
The directory structure of spark similar to hadoop 2.3.0.
There are three subcomponents: core,examples,hive,The directory structure of
spark:
{code}
+- SPARK_HOME
| +- bin
| \- *
| +- sbin
| \- *
| +- RELEASE
| +- conf
| \- *
| +- python
| \- *
| +- share
| \- spark
|+- core
| +- lib
| \- *.jar
| +- spark-core*.jar
| +- spark-repl*.jar
| +- spark-yarn*.jar
| +- spark-bagel*.jar
| +- spark-graphx*.jar
| +- spark-sql*.jar
| +- spark-catalyst*.jar
| +- spark-mllib*.jar
| \- spark-streaming*.jar
|+- hive
| +- lib
| \- *.jar
| \- spark-hive*.jar
|+- examples
| +- lib
| \- *.jar
| \- spark-examples*.jar
{code}
Improve spark integration
-
Key: SPARK-1698
URL: https://issues.apache.org/jira/browse/SPARK-1698
Project: Spark
Issue Type: Improvement
Components: Build, Deploy
Reporter: Guoqiang Li
Assignee: Guoqiang Li
Fix For: 1.0.0
Use the shade plugin to create a big JAR with all the dependencies can cause
a few problems
1. Missing jar's meta information
2. Some file is covered, eg: plugin.xml
3. Different versions of a jar may co-exist
4. Too big, java 6 does not support
--
This message was sent by Atlassian JIRA
(v6.2#6252)
