cloud-fan commented on a change in pull request #26806: [SPARK-30183][SQL]
Disallow to specify reserved properties in CREATE NAMESPACE syntax
URL: https://github.com/apache/spark/pull/26806#discussion_r363626739
##########
File path: docs/sql-migration-guide.md
##########
@@ -264,6 +264,8 @@ license: |
- Since Spark 3.0, the function `percentile_approx` and its alias
`approx_percentile` only accept integral value with range in `[1, 2147483647]`
as its 3rd argument `accuracy`, fractional and string types are disallowed,
e.g. `percentile_approx(10.0, 0.2, 1.8D)` will cause `AnalysisException`. In
Spark version 2.4 and earlier, if `accuracy` is fractional or string value, it
will be coerced to an int value, `percentile_approx(10.0, 0.2, 1.8D)` is
operated as `percentile_approx(10.0, 0.2, 1)` which results in `10.0`.
+ - Since Spark 3.0, the namespace properties `location` and `comment` become
reserved, it will fail with `ParseException` if we use them as members of
`DBPROTERTIES` in `CREATE NAMESPACE` and `ALTER NAMESPACE ... SET
PROPERTIES(...)`. We need their specific clauses to specify them, e.g. `CREATE
NAMESPACE a.b.c COMMENT 'any comment' LOCATION 'some path'`. We can set
`spark.sql.legacy.property.nonReserved` to `true` to ignore the
`ParseException`, but notice that in this case, these properties will produce
side effects, e.g `SET DBPROTERTIES('location'='/tmp')` might change the
location of the database. In Spark version 2.4 and earlier, these properties
are neither reserved nor have side effects, e.g. `SET
DBPROTERTIES('location'='/tmp')` will not change the location of the database
but create a headless property just like `'a'='b'`.
Review comment:
Since it's the migration guide, I think it's better to use the syntax of
Spark 2.4
```
it will fail with `ParseException` if we use them in `CREATE DATABASE ...
DBPROPERTIES` and `ALTER DATABASE ... SET DBPROPERTIES` ...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
