[GitHub] spark pull request #20702: [SPARK-23547][SQL]Cleanup the .pipeout file when ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/20702


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20702: [SPARK-23547][SQL]Cleanup the .pipeout file when ...

[zuotingbing]

Github user zuotingbing commented on a diff in the pull request:

https://github.com/apache/spark/pull/20702#discussion_r172166003
  
--- Diff: 
sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java
 ---
@@ -665,6 +667,25 @@ public void close() throws HiveSQLException {
 }
   }
 
+  private void cleanupPipeoutFile() {
+String lScratchDir = hiveConf.getVar(ConfVars.LOCALSCRATCHDIR);
+String sessionID = hiveConf.getVar(ConfVars.HIVESESSIONID);
+
+File[] fileAry = new File(lScratchDir).listFiles(
--- End diff --

@attilapiros  will fix it, thanks!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20702: [SPARK-23547][SQL]Cleanup the .pipeout file when ...

[attilapiros]

Github user attilapiros commented on a diff in the pull request:

https://github.com/apache/spark/pull/20702#discussion_r172003781
  
--- Diff: 
sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java
 ---
@@ -665,6 +667,25 @@ public void close() throws HiveSQLException {
 }
   }
 
+  private void cleanupPipeoutFile() {
+String lScratchDir = hiveConf.getVar(ConfVars.LOCALSCRATCHDIR);
+String sessionID = hiveConf.getVar(ConfVars.HIVESESSIONID);
+
+File[] fileAry = new File(lScratchDir).listFiles(
--- End diff --

No problem. I hope this works.

```
File[] fileAry = new File(lScratchDir).listFiles(
  (dir, name) -> name.startsWith(sessionID) && 
name.endsWith(".pipeout"));
```

I think it would be good if you would have one unit test for your change.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20702: [SPARK-23547][SQL]Cleanup the .pipeout file when ...

[zuotingbing]

Github user zuotingbing commented on a diff in the pull request:

https://github.com/apache/spark/pull/20702#discussion_r171749296
  
--- Diff: 
sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java
 ---
@@ -665,6 +667,25 @@ public void close() throws HiveSQLException {
 }
   }
 
+  private void cleanupPipeoutFile() {
+String lScratchDir = hiveConf.getVar(ConfVars.LOCALSCRATCHDIR);
+String sessionID = hiveConf.getVar(ConfVars.HIVESESSIONID);
+
+File[] fileAry = new File(lScratchDir).listFiles(
--- End diff --

Sorry i did not find the overloaded version of listfiles which saves 1 
line. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20702: [SPARK-23547][SQL]Cleanup the .pipeout file when ...

[attilapiros]

Github user attilapiros commented on a diff in the pull request:

https://github.com/apache/spark/pull/20702#discussion_r171668166
  
--- Diff: 
sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java
 ---
@@ -665,6 +667,25 @@ public void close() throws HiveSQLException {
 }
   }
 
+  private void cleanupPipeoutFile() {
+String lScratchDir = hiveConf.getVar(ConfVars.LOCALSCRATCHDIR);
+String sessionID = hiveConf.getVar(ConfVars.HIVESESSIONID);
+
+File[] fileAry = new File(lScratchDir).listFiles(
--- End diff --

Nit: There is an overloaded version of 
[listfiles](https://docs.oracle.com/javase/8/docs/api/java/io/File.html#listFiles-java.io.FilenameFilter-).
Using it saves 1 line.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20702: [SPARK-23547][SQL]Cleanup the .pipeout file when ...

[zuotingbing]

GitHub user zuotingbing opened a pull request:

https://github.com/apache/spark/pull/20702

[SPARK-23547][SQL]Cleanup the .pipeout file when the Hive Session closed

## What changes were proposed in this pull request?

when the hive session closed, we should also cleanup the .pipeout file.

## How was this patch tested?

 manual tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zuotingbing/spark SPARK-23547

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20702.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20702


commit d965ba2b2ea34a8a0a8aef86a731e01c4d1b4ac1
Author: zuotingbing 
Date:   2018-03-01T12:18:18Z

[SPARK-23547][SQL]Cleanup the .pipeout file when the Hive Session closed




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org