[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530383#comment-15530383
]
Imran Rashid commented on HBASE-16179:
--------------------------------------
comments on v9:
* In JavaHBaseContext.scala
{code}
- def fn = (it: Iterator[T], conn: Connection) =>
- asScalaIterator(
- f.call((asJavaIterator(it), conn)).iterator()
- )
+ def fn = (it: Iterator[T], conn: Connection) => {
+ val iter = f.call(asJavaIterator(it), conn)
+ if (iter.isInstanceOf[Iterable[R]]) {
+ asScalaIterator(iter.asInstanceOf[Iterable[R]].iterator())
+ } else {
+ asScalaIterator(iter)
+ }
+ }
{code}
why do you need the if instanceOf? {{f.call}} will always return an iterator,
isn't that all you need? I thought you'd just remove the {{.iterator()}} in
the original version.
* In HBaseTableScanRDD.scala you are removing the mixed in {{Logging}} trait.
But there are still calls to {{logDebug}} etc. How does that work? Seems like
its using the logging from within RDD, but I thought the scala compiler would
complain about escaped visibility scope or something. I would think you'd need
to either remove the calls to {{logDebug}}, or do something like this to use
the hbase Logging trait:
{code}
class HBaseTableScanRDD ... {
val logger = new Object with Logging
...
logger.logDebug(...)
}
{code}
I'd be curious if it really does work like this, it would mean spark is leaking
something it doesn't intend to.
* I would think that the spark & scala version properties in
hbase-spark/pom.xml would be set inside the profiles, but I admit I don't spend
a ton of time following the exact pom setup, so maybe its fine.
* a couple of commented out bits that should be removed
> Fix compilation errors when building hbase-spark against Spark 2.0
> ------------------------------------------------------------------
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
> Issue Type: Bug
> Components: spark
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt,
> 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
