[
https://issues.apache.org/jira/browse/SPARK-11855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014180#comment-15014180
]
Michael Armbrust commented on SPARK-11855:
------------------------------------------
Hey, very sorry for the confusion here. Catalyst is not a public API and is
explicitly hidden from the public documentation and MIMA for this reason. We
also document this fact
[here|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/package.scala].
I don't know of any database that exposes this level of detail about the
inner workings in a stable way.
We avoid using package protection so that developers can work with the library
in the REPL and so that universities can use Spark as a platform for doing
research. However, in order to keep pushing the public APIs forward, we have
to reserve the right to change internals.
A compromise I'm totally willing to accept would be to add deprecated
compatibility methods for at least a couple of versions, provided the community
is willing to put in the effort to make PR adding them.
> Catalyst breaks backwards compatibility in branch-1.6
> -----------------------------------------------------
>
> Key: SPARK-11855
> URL: https://issues.apache.org/jira/browse/SPARK-11855
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.0
> Reporter: Santiago M. Mola
> Priority: Critical
>
> There's a number of APIs broken in catalyst 1.6.0. I'm trying to compile most
> cases:
> *UnresolvedRelation*'s constructor has been changed from taking a Seq to a
> TableIdentifier. A deprecated constructor taking Seq would be needed to be
> backwards compatible.
> {code}
> case class UnresolvedRelation(
> - tableIdentifier: Seq[String],
> + tableIdentifier: TableIdentifier,
> alias: Option[String] = None) extends LeafNode {
> {code}
> It is similar with *UnresolvedStar*:
> {code}
> -case class UnresolvedStar(table: Option[String]) extends Star with
> Unevaluable {
> +case class UnresolvedStar(target: Option[Seq[String]]) extends Star with
> Unevaluable {
> {code}
> *Catalog* did get a lot of signatures changed too (because of
> TableIdentifier). Providing the older methods as deprecated also seems viable
> here.
> Spark 1.5 already broke backwards compatibility of part of catalyst API with
> respect to 1.4. I understand there are good reasons for some cases, but we
> should try to minimize backwards compatibility breakages for 1.x. Specially
> now that 2.x is on the horizon and there will be a near opportunity to remove
> deprecated stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
