[ https://issues.apache.org/jira/browse/SPARK-11855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014180#comment-15014180 ]
Michael Armbrust commented on SPARK-11855: ------------------------------------------ Hey, very sorry for the confusion here. Catalyst is not a public API and is explicitly hidden from the public documentation and MIMA for this reason. We also document this fact [here|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/package.scala]. I don't know of any database that exposes this level of detail about the inner workings in a stable way. We avoid using package protection so that developers can work with the library in the REPL and so that universities can use Spark as a platform for doing research. However, in order to keep pushing the public APIs forward, we have to reserve the right to change internals. A compromise I'm totally willing to accept would be to add deprecated compatibility methods for at least a couple of versions, provided the community is willing to put in the effort to make PR adding them. > Catalyst breaks backwards compatibility in branch-1.6 > ----------------------------------------------------- > > Key: SPARK-11855 > URL: https://issues.apache.org/jira/browse/SPARK-11855 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.6.0 > Reporter: Santiago M. Mola > Priority: Critical > > There's a number of APIs broken in catalyst 1.6.0. I'm trying to compile most > cases: > *UnresolvedRelation*'s constructor has been changed from taking a Seq to a > TableIdentifier. A deprecated constructor taking Seq would be needed to be > backwards compatible. > {code} > case class UnresolvedRelation( > - tableIdentifier: Seq[String], > + tableIdentifier: TableIdentifier, > alias: Option[String] = None) extends LeafNode { > {code} > It is similar with *UnresolvedStar*: > {code} > -case class UnresolvedStar(table: Option[String]) extends Star with > Unevaluable { > +case class UnresolvedStar(target: Option[Seq[String]]) extends Star with > Unevaluable { > {code} > *Catalog* did get a lot of signatures changed too (because of > TableIdentifier). Providing the older methods as deprecated also seems viable > here. > Spark 1.5 already broke backwards compatibility of part of catalyst API with > respect to 1.4. I understand there are good reasons for some cases, but we > should try to minimize backwards compatibility breakages for 1.x. Specially > now that 2.x is on the horizon and there will be a near opportunity to remove > deprecated stuff. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org