[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530546#comment-15530546
]
Imran Rashid commented on HBASE-16179:
--------------------------------------
ah, sorry I forgot that you were making JavaHBaseContext.scala work with both
versions of spark. I'd at least include a comment to that effect. As is, it
chooses the branch at runtime, as opposed to choosing which one at compile time
by having two different implementations which get chosen by the profile.
(Doesn't really matter to me which one you do, just noting the difference.) A
minor nit, if you are going to stick w/ doing it at runtime, I'd write it with
pattern matching (this may also fix the compile errors w/ 1.6):
{code}
def fn = (it: Iterator[T], conn: Connection) => {
asScalaIterator(
// the return type is different in spark 1.x & 2.x, we handle both cases
f.call(asJavaIterator(it), conn) match {
// spark 1.x
case iterable: Iterable[R] => iterable.iterator()
// spark 2.x
case iterator: Iterator[R] => iterator
}
)
}
{code}
on HBaseTableScanRDD -- I am surprised it can use the logging methods from
spark's internal Logging, thought that was against scala's scoping rules. But
I guess I was wrong if it works.
> Fix compilation errors when building hbase-spark against Spark 2.0
> ------------------------------------------------------------------
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
> Issue Type: Bug
> Components: spark
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt,
> 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
