sunchao commented on code in PR #3831:
URL: https://github.com/apache/hive/pull/3831#discussion_r1040208683
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:
##########
@@ -1971,13 +1971,19 @@ public static Path createMoveTask(Task<?> currTask,
boolean chDir,
* 2. INSERT operation on full ACID table
*/
if (!isMmTable && !isDirectInsert) {
- // generate the temporary file
- // it must be on the same file system as the current destination
Context baseCtx = parseCtx.getContext();
- // Create the required temporary file in the HDFS location if the
destination
- // path of the FileSinkOperator table is a blobstore path.
- Path tmpDir =
baseCtx.getTempDirForFinalJobPath(fileSinkDesc.getDestPath());
+ // Choose location of required temporary file
+ Path tmpDir = null;
+ if (hconf.getBoolVar(ConfVars.HIVE_USE_SCRATCHDIR_FOR_STAGING)) {
+ tmpDir =
baseCtx.getTempDirForInterimJobPath(fileSinkDesc.getDestPath());
+ } else {
+ tmpDir =
baseCtx.getTempDirForFinalJobPath(fileSinkDesc.getDestPath());
+ }
+ DynamicPartitionCtx dpCtx = fileSinkDesc.getDynPartCtx();
+ if (dpCtx != null && dpCtx.getSPPath() != null) {
+ tmpDir = new Path(tmpDir, dpCtx.getSPPath());
Review Comment:
nit: 2 space indentation
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -5629,6 +5629,10 @@ public static enum ConfVars {
"This is a performance optimization that forces the final
FileSinkOperator to write to the blobstore.\n" +
"See HIVE-15121 for details."),
+ HIVE_USE_SCRATCHDIR_FOR_STAGING("hive.use.scratchdir.for.staging", false,
+ "Use ${hive.exec.scratchdir} for query results instead of
${hive.exec.stagingdir}.\n" +
+ "This stages query results in ${hive.exec.scratchdir} before move
to final destination."),
Review Comment:
nit: move -> moving
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##########
@@ -2608,8 +2608,8 @@ private Partition loadPartitionInternal(Path loadPath,
Table tbl, Map<String, St
* See: HIVE-1707 and HIVE-2117 for background
*/
FileSystem oldPartPathFS = oldPartPath.getFileSystem(getConf());
- FileSystem loadPathFS = loadPath.getFileSystem(getConf());
- if (FileUtils.equalsFileSystem(oldPartPathFS,loadPathFS)) {
+ FileSystem tblPathFS = tblDataLocationPath.getFileSystem(getConf());
+ if (FileUtils.equalsFileSystem(oldPartPathFS,tblPathFS)) {
Review Comment:
nit: space before `tblPathFs`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
