[
https://issues.apache.org/jira/browse/HBASE-18175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053466#comment-16053466
]
Yi Liang edited comment on HBASE-18175 at 6/19/17 4:28 AM:
-----------------------------------------------------------
Hi Sean,
In the new patch, I add a submodule called hbase-spark-it under hbase-spark
module, the parent of hbase-spark-it is hbase-spark, not hbase, this can make
hbase-spark module more independent.
Now the hbase-spark looks like
---hbase-spark
-----------src
-----------pom
-----------hbase-spark-it
---------------src
---------------pom
And I also have some thoughts on HBase-16179, only solve that jira then we can
push hbase-spark into hbase-2.0
And I saw your comments in that jira, there might be some requirement like, we
need to consider spark-1.6, spark-2.0, and also scala 2.10, scala 2.11
I remember I try both the scala 2.10 and 2.11 for hbase-spark, it seems that
there is no code change for scala version change; the code differences are only
in different spark versions.
I think we can have hbase-spark module structure as below
- hbase-spark
--------hbase-spark-1.6
------------src (diff code for 1.6)
------------pom
--------hbase-spark-2.0
--------hbase-spark-it
------------src
------------pom
And user can change scala version on their own for both spark1.6 and spark2.0,
What do you think of this idea?
Thanks
was (Author: easyliangjob):
Hi Sean,
In the new patch, I add a submodule called hbase-spark-it under hbase-spark
module, the parent of hbase-spark-it is hbase-spark, not hbase, this can make
hbase-spark module more independent.
Now the hbase-spark looks like
- hbase-spark
-hbase-spark-it
-src
-pom
-src
-pom
And I also have some thoughts on HBase-16179, only solve that jira then we can
push hbase-spark into hbase-2.0
And I saw your comments in that jira, there might be some requirement like, we
need to consider spark-1.6, spark-2.0, and also scala 2.10, scala 2.11
I remember I try both the scala 2.10 and 2.11 for hbase-spark, it seems that
there is no code change for scala version change; the code differences are only
in different spark versions.
I think we can have hbase-spark module structure as below
- hbase-spark
-hbase-spark-1.6
-src (diff code for 1.6)
-pom
-hbase-spark-2.0
-hbase-spark-it
-src
-pom
And user can change scala version on their own for both spark1.6 and spark2.0,
What do you think of this idea?
Thanks
> Add hbase-spark integration test into hbase-it
> ----------------------------------------------
>
> Key: HBASE-18175
> URL: https://issues.apache.org/jira/browse/HBASE-18175
> Project: HBase
> Issue Type: Test
> Components: spark
> Reporter: Yi Liang
> Assignee: Yi Liang
> Priority: Critical
> Fix For: 2.0.0
>
> Attachments: hbase-18175-v1.patch
>
>
> After HBASE-17574, all test under hbase-spark are regarded as unit test, and
> this jira will add integration test of hbase-spark into hbase-it. This patch
> run same tests as mapreduce.IntegrationTestBulkLoad, just change mapreduce to
> spark.
> test in Maven:
> mvn verify -Dit.test=IntegrationTestSparkBulkLoad
> test on cluster:
> spark-submit --class
> org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad
> HBASE_HOME/lib/hbase-it-2.0.0-SNAPSHOT-tests.jar
> -Dhbase.spark.bulkload.chainlength=500000 -m slowDeterministic
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
