yhuai commented on a change in pull request #27670: [MINOR][DOC] Add two
migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#discussion_r382697622
##########
File path: docs/sql-migration-guide.md
##########
@@ -335,6 +335,10 @@ license: |
- Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. You need
to migrate your custom SerDes to Hive 2.3 or build your own Spark with
`hive-1.2` profile. See HIVE-15167 for more details.
+ - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The result
decimal represent in string can be different between Hive 1.2 and Hive 2.3 when
use `TRANSFORM` syntax in sql. In Hive 1.2, decimal will eliminate all
non-significant zeros in precision when represents in string. But in Hive 2.3,
it will always show 18 digits in precision and make up with zero if its
precision is less than 18.
+
+ - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The local
file system has changed from `org.apache.hadoop.fs.LocalFileSystem` to
`org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem` when use Hadoop API, e.g.
`new Path("file:/").getFileSystem(conf)`, and `fs.rename(Path src, Path dst)`
will returns `true` in Hive 1.2 but `false` in Hive 2.3 if `dst` represents a
file.
Review comment:
Should we group all of hive related behavior change under the same section
or the same bullet point? Then, we make individual changes sub-bullet-points.
Also, when we say rename returns true for hive 1.2 when des exists, we
should call out that the destination will get overwritten, right? Also, can we
link to related hadoop and hive tickets?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
