GitHub user fjh100456 opened a pull request:
https://github.com/apache/spark/pull/22412
[SPARK-25404][SQL] Staging path may not on the expected place when table
path contains the stagingDir string
## What changes were proposed in this pull request?
As described in
[#SPARK-25404](https://issues.apache.org/jira/browse/SPARK-25404), staging
path may not on the right place we expect. I'm not quiet sure in which case
the `inputPathName` contains the `stagingDir`, but it seems `new
Path(inputPathName, stagingDir).toString` is enough.
```scala
var stagingPathName: String =
if (inputPathName.indexOf(stagingDir) == -1) {
new Path(inputPathName, stagingDir).toString
} else {
inputPathName.substring(0, inputPathName.indexOf(stagingDir) +
stagingDir.length)
}
```
## How was this patch tested?
Manually test with debug mode, and check the staging files on right path.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fjh100456/spark master-StagingDir
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22412.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22412
----
commit a6d48558111b6968919a692f6b2d8a9095fc64a4
Author: fjh100456 <fu.jinhua6@...>
Date: 2018-09-13T11:28:12Z
[SPARK-25404][SQL] Staging path may not on the expected place when table
path contains the stagingDir string
## What changes were proposed in this pull request?
Fix Staging path.
## How was this patch tested?
Manual test.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
