[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[
https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758619#comment-16758619
]
Hadoop QA commented on HBASE-18405:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HBASE-18405 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.8.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-18405 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15839/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Track scope for HBase-Spark module
> --
>
> Key: HBASE-18405
> URL: https://issues.apache.org/jira/browse/HBASE-18405
> Project: HBase
> Issue Type: Task
> Components: spark
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 1.5.1
>
> Attachments: Apache HBase - Apache Spark Integration Scope - update
> 1.pdf, Apache HBase - Apache Spark Integration Scope.pdf
>
>
> Start with [\[DISCUSS\] status of and plans for our hbase-spark integration
> |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E]
> and formalize into a scope document for bringing this feature into a release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678479#comment-16678479 ] stack commented on HBASE-18405: --- Fixed. Sorry about that. > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: Apache HBase - Apache Spark Integration Scope - update > 1.pdf, Apache HBase - Apache Spark Integration Scope.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678472#comment-16678472 ] Josh Elser commented on HBASE-18405: Permissions not open on the doc? Requested access :) > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: Apache HBase - Apache Spark Integration Scope - update > 1.pdf, Apache HBase - Apache Spark Integration Scope.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[
https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677361#comment-16677361
]
Hadoop QA commented on HBASE-18405:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HBASE-18405 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.8.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-18405 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14968/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Track scope for HBase-Spark module
> --
>
> Key: HBASE-18405
> URL: https://issues.apache.org/jira/browse/HBASE-18405
> Project: HBase
> Issue Type: Task
> Components: spark
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: Apache HBase - Apache Spark Integration Scope - update
> 1.pdf, Apache HBase - Apache Spark Integration Scope.pdf
>
>
> Start with [\[DISCUSS\] status of and plans for our hbase-spark integration
> |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E]
> and formalize into a scope document for bringing this feature into a release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677357#comment-16677357 ] stack commented on HBASE-18405: --- Doc in which to keep notes on projects and objectives touched-on long ago when this JIRA started a year and a half ago. > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: Apache HBase - Apache Spark Integration Scope - update > 1.pdf, Apache HBase - Apache Spark Integration Scope.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677355#comment-16677355 ] stack commented on HBASE-18405: --- Trying to revive this project. Holding to the original scope doc except for fact that hbase-spark* modules get moved to new hbase-connectors repo. Let me attach doc that tries to revisit the points listed in the thread cited above at the head of this issue subsequently realized in the attached scope doc. > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: Apache HBase - Apache Spark Integration Scope - update > 1.pdf, Apache HBase - Apache Spark Integration Scope.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182803#comment-16182803 ] Sean Busbey commented on HBASE-18405: - I've been using it as an umbrella for the hbase-spark module being ready to ship. So we'd need related jiras for all the gaps between the current implementation in master and the scope doc. then we'd need all related jiras closed. > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 3.0.0, 2.1.0, 1.5.0 > > Attachments: Apache HBase - Apache Spark Integration Scope.pdf, > Apache HBase - Apache Spark Integration Scope - update 1.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181623#comment-16181623 ] Mike Drob commented on HBASE-18405: --- What do we need to mark this done, [~busbey]? > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 3.0.0, 2.1.0, 1.5.0 > > Attachments: Apache HBase - Apache Spark Integration Scope.pdf, > Apache HBase - Apache Spark Integration Scope - update 1.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119192#comment-16119192 ] Weiqing Yang commented on HBASE-18405: -- A late LGTM. Thanks, [~busbey] :) > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 1.4.0, 2.0.0-beta-1 > > Attachments: Apache HBase - Apache Spark Integration Scope.pdf, > Apache HBase - Apache Spark Integration Scope - update 1.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115969#comment-16115969 ] stack commented on HBASE-18405: --- Read update1. Excellent writeup [~busbey] Non-important thoughts: + Composite row key is spark being able to grok a preexisting hbase row key; i.e. teaching spark how to extract subparts. How would this be done in the future do you think? + No typing then in the first pass. Ain't that going to be ugly perf-wise sir? Downsides? How you see the project that does the smooth mapping of spark to hbase-type (whether phoenix or hbase data type). + What is this one about... "... multiple secure HBase deployments"? A spark job spanning hbase clusters? Avro? Because? You can do schema apart from data? Java Native is Bytes.toLong, etc.? You looked at providing a 'catalog'? Would it be tough? Could we ship a hard-coded table with type info in it with a facade that implements catalog to satisfy spark? Or would folks be needing to extend with their own compound types, etc. bq. Where practical, we will avoid duplication of implementation source code.. Is this like our current hadoop-compatible for metrics, etc., with hadoop 2 and hadoop1 implementations? On unit tests, will we have to spin up clusters? Can we get away with things like the RegionAsTable dohickey (puts a Table Interface on a Region...). You know what I'm worried about ... You start the build when you leave on your two-week vacation and you hope it is done when you get back. Spark in a new project like hbase-thirdparty? Great writeup. > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 1.4.0, 2.0.0-beta-1 > > Attachments: Apache HBase - Apache Spark Integration Scope.pdf, > Apache HBase - Apache Spark Integration Scope - update 1.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102056#comment-16102056 ] Sean Busbey commented on HBASE-18405: - excellent. Let's go ahead and call this scope set. I'll get the relevant JIRAs for our gaps linked later this week. > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 1.4.0, 2.0.0-beta-1 > > Attachments: Apache HBase - Apache Spark Integration Scope.pdf, > Apache HBase - Apache Spark Integration Scope - update 1.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098901#comment-16098901 ] Yi Liang commented on HBASE-18405: -- Hi Sean, Update1 looks good to me, I think it has cover most areas for integration of hbase-spark. +1 moving forward with this as a guide > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 1.4.0, 2.0.0-beta-1 > > Attachments: Apache HBase - Apache Spark Integration Scope.pdf, > Apache HBase - Apache Spark Integration Scope - update 1.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096902#comment-16096902 ] Sean Busbey commented on HBASE-18405: - Any feedback here folks? Objections to us moving forward with this as a guide? ping [~easyliangjob], [~jerryhe], [~Weiqing Yang], [~tedyu] > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 1.4.0, 2.0.0-beta-1 > > Attachments: Apache HBase - Apache Spark Integration Scope.pdf, > Apache HBase - Apache Spark Integration Scope - update 1.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093705#comment-16093705 ] Sean Busbey commented on HBASE-18405: - glad to hear it. thanks again for the review! > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 1.4.0, 2.0.0-beta-1 > > Attachments: Apache HBase - Apache Spark Integration Scope.pdf, > Apache HBase - Apache Spark Integration Scope - update 1.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093646#comment-16093646 ] Mike Drob commented on HBASE-18405: --- Thanks for addressing my comments. Update 1 looks good to me. > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 1.4.0, 2.0.0-beta-1 > > Attachments: Apache HBase - Apache Spark Integration Scope.pdf, > Apache HBase - Apache Spark Integration Scope - update 1.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[
https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092241#comment-16092241
]
Sean Busbey commented on HBASE-18405:
-
{quote}
bq. I thought that's what I read. I also thought works-correctly-enough made it
into 2.0. lemme go jira hunting.
SPARK-14743 is the one I'm thinking of.
{quote}
SPARK-12523 works well enough for us to still do 2.0, IMO. If we want feature
parity with SHC then we'd need to add a similar credential manager, which will
require SPARK-14743, but that all gets configured outside of the spark
application AFAICT.
> Track scope for HBase-Spark module
> --
>
> Key: HBASE-18405
> URL: https://issues.apache.org/jira/browse/HBASE-18405
> Project: HBase
> Issue Type: Task
> Components: spark
>Reporter: Sean Busbey
>Assignee: Sean Busbey
> Fix For: 1.4.0, 2.0.0-beta-1
>
> Attachments: Apache HBase - Apache Spark Integration Scope.pdf
>
>
> Start with [\[DISCUSS\] status of and plans for our hbase-spark integration
> |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E]
> and formalize into a scope document for bringing this feature into a release.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[
https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092229#comment-16092229
]
Sean Busbey commented on HBASE-18405:
-
{quote}
bq. avoid a "we'll do it later" that turns into not getting done. Worth the
trouble?
If you mark it a blocker for 2.0, it should get done, no?
{quote}
I want it as a blocker for the branch-1 release the backport goes to though.
{quote}
bq. the integration test already exists, so I didn't keep it in a milestone
(barring something showing up in milestone 0's results).
One integration test exists. Do you think this provides sufficient coverage?
{quote}
My big concern in the integration test is that we can get loaded by spark and
eventually that we work with their delegation tokens. One test will provide
coverage for that (presuming someone actually runs it on a cluster).
bq. Would love to see the IT get run with monkeys also, not sure if it
currently supports that.
AFAIK there's nothing in the IT that would prevent it running with monkeys as
the other tests do. Unfortunately none of our ITs run with monkeys on
build.a.o, so I wouldn't want to make that a blocker.
bq. Scaladoc update is missing from milestones. Likely M4.
d'oh. I'll add that as a proper milestone in the next rev, esp since it might
mean a website update.
> Track scope for HBase-Spark module
> --
>
> Key: HBASE-18405
> URL: https://issues.apache.org/jira/browse/HBASE-18405
> Project: HBase
> Issue Type: Task
> Components: spark
>Reporter: Sean Busbey
>Assignee: Sean Busbey
> Fix For: 1.4.0, 2.0.0-beta-1
>
> Attachments: Apache HBase - Apache Spark Integration Scope.pdf
>
>
> Start with [\[DISCUSS\] status of and plans for our hbase-spark integration
> |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E]
> and formalize into a scope document for bringing this feature into a release.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[
https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092214#comment-16092214
]
Sean Busbey commented on HBASE-18405:
-
{quote}
If we have multiple modules for a cartesian product of spark and scala
versions, do we need corresponding IT modules for each of them? Can we run the
same ITs several times in a matrix within maven? Do we want to reserve that for
a profile? What do we expect daily developers to do, what do we expect RMs to
do?
{quote}
oh! okay yeah that's a big gap. Thanks!
> Track scope for HBase-Spark module
> --
>
> Key: HBASE-18405
> URL: https://issues.apache.org/jira/browse/HBASE-18405
> Project: HBase
> Issue Type: Task
> Components: spark
>Reporter: Sean Busbey
>Assignee: Sean Busbey
> Fix For: 1.4.0, 2.0.0-beta-1
>
> Attachments: Apache HBase - Apache Spark Integration Scope.pdf
>
>
> Start with [\[DISCUSS\] status of and plans for our hbase-spark integration
> |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E]
> and formalize into a scope document for bringing this feature into a release.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092207#comment-16092207 ] Mike Drob commented on HBASE-18405: --- If we have multiple modules for a cartesian product of spark and scala versions, do we need corresponding IT modules for each of them? Can we run the same ITs several times in a matrix within maven? Do we want to reserve that for a profile? What do we expect daily developers to do, what do we expect RMs to do? bq. I thought that's what I read. I also thought works-correctly-enough made it into 2.0. lemme go jira hunting. SPARK-14743 is the one I'm thinking of. bq. avoid a "we'll do it later" that turns into not getting done. Worth the trouble? If you mark it a blocker for 2.0, it should get done, no? bq. the integration test already exists, so I didn't keep it in a milestone (barring something showing up in milestone 0's results). _One_ integration test exists. Do you think this provides sufficient coverage? Would love to see the IT get run with monkeys also, not sure if it currently supports that. bq. Maybe add a milestone after 4 that's "ensure tests on asf jenkins" that includes the precommit updates? Didn't you already do this? Scaladoc update is missing from milestones. Likely M4. > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 1.4.0, 2.0.0-beta-1 > > Attachments: Apache HBase - Apache Spark Integration Scope.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092186#comment-16092186 ] Sean Busbey commented on HBASE-18405: - thanks for the quick turn around time on review! bq. Module Layout should address hbase-spark-it module, I think. I tried to get this across without specifying the actual modules used in implementation. It meets all the requirements: is a new module, depends on old modules, only shows up as a reactor module and in the assembly. Or do you mean I need to call out that the test stuff is subject to the same constraints? Should I just go ahead an prescribe a layout? bq. Did we actaully determine on the DISCUSS thread that we must support Spark 2.0+? We might have to take a 2.1+ position, since I think there was some updates to work with Delegation Tokens that didn't make it in right away and another goal is to have use case parity for secure deployments. I thought that's what I read. I also thought works-correctly-enough made it into 2.0. lemme go jira hunting. bq. Milestone 5 backport could happen before Milestone 4 documentation is done. You said that docs get updated closer to release time anyway in another thread on list. Good point. I think the main reason I would like to list it first is to avoid a "we'll do it later" that turns into not getting done. Worth the trouble? bq. Integration Tests and Yetus tests are not included in any milestones. Combine with Unit Test M2? the integration test already exists, so I didn't keep it in a milestone (barring something showing up in milestone 0's results). milestone 1 covers the yetus changes, though it occurs to me now that it references docs that won't exist until milestone 4. Milestone 2 I meant to cover "write tests" basically. Maybe add a milestone after 4 that's "ensure tests on asf jenkins" that includes the precommit updates? > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 1.4.0, 2.0.0-beta-1 > > Attachments: Apache HBase - Apache Spark Integration Scope.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092135#comment-16092135 ] Mike Drob commented on HBASE-18405: --- Module Layout should address hbase-spark-it module, I think. Did we actaully determine on the DISCUSS thread that we must support Spark 2.0+? We might have to take a 2.1+ position, since I think there was some updates to work with Delegation Tokens that didn't make it in right away and another goal is to have use case parity for secure deployments. Milestone 1 summary is missing a word at the end? Milestone 5 backport could happen before Milestone 4 documentation is done. You said that docs get updated closer to release time anyway in another thread on list. Integration Tests and Yetus tests are not included in any milestones. Combine with Unit Test M2? > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 1.4.0, 2.0.0-beta-1 > > Attachments: Apache HBase - Apache Spark Integration Scope.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
