cloud-fan commented on a change in pull request #28026:
URL: https://github.com/apache/spark/pull/28026#discussion_r513219480
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
##########
@@ -295,18 +295,61 @@ private[sql] object CatalogV2Util {
catalog.name().equalsIgnoreCase(CatalogManager.SESSION_CATALOG_NAME)
}
- def convertTableProperties(
+ def convertTableProperties(c: CreateTableStatement): Map[String, String] = {
+ convertTableProperties(
+ c.properties, c.options, c.serde, c.location, c.comment, c.provider,
c.external)
+ }
+
+ def convertTableProperties(c: CreateTableAsSelectStatement): Map[String,
String] = {
+ convertTableProperties(
+ c.properties, c.options, c.serde, c.location, c.comment, c.provider,
c.external)
+ }
+
+ def convertTableProperties(r: ReplaceTableStatement): Map[String, String] = {
+ convertTableProperties(r.properties, r.options, r.serde, r.location,
r.comment, r.provider)
+ }
+
+ def convertTableProperties(r: ReplaceTableAsSelectStatement): Map[String,
String] = {
+ convertTableProperties(r.properties, r.options, r.serde, r.location,
r.comment, r.provider)
+ }
+
+ private def convertTableProperties(
properties: Map[String, String],
options: Map[String, String],
+ serdeInfo: Option[SerdeInfo],
location: Option[String],
comment: Option[String],
- provider: Option[String]): Map[String, String] = {
- properties ++ options ++
+ provider: Option[String],
+ external: Boolean = false): Map[String, String] = {
+ properties ++
+ options ++ // to make the transition to the "option." prefix easier, add
both
+ options.map { case (key, value) => TableCatalog.OPTION_PREFIX + key ->
value } ++
+ convertToProperties(serdeInfo) ++
+ (if (external) Map(TableCatalog.PROP_EXTERNAL -> "true") else Map.empty)
++
Review comment:
@RussellSpitzer We use JIRA tickets to generate release notes. It's hard
for the release manager to tell the API/behavior changes I mentioned earlier
from this PR title "Unify create table syntax".
Also as @HyukjinKwon mentioned, smaller PRs can help to narrow down the
discussion scope. This PR is hard for tracking purposes as the discussion is so
long. Splitting this PR won't help too much now as the discussion is already
done, but it can still help code review.
My baseline is to create separated JIRA tickets, to mention the decisions
explicitly. For example, in this PR, `OPTIONS ...` will appear twice in the
table properties, once with `option.` prefix, once without.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
