Github user Sephiroth-Lin commented on the issue:
https://github.com/apache/spark/pull/13651
LGTM thank you
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user Sephiroth-Lin commented on the issue:
https://github.com/apache/spark/pull/13678
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user Sephiroth-Lin commented on the issue:
https://github.com/apache/spark/pull/13524
@rxin Done. Pleas help review, thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/13561
[SPARK-15824][SQL] Run 'with ... insert ... select' failed when use spark
thriftserver
## What changes were proposed in this pull request?
Dataset.collect will call
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/13524
[SPARK-15776] Type coercion incorrect
## What changes were proposed in this pull request?
Update type coercion order, details see
https://issues.apache.org/jira/browse/SPARK-15776
Github user Sephiroth-Lin closed the pull request at:
https://github.com/apache/spark/pull/7417
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/7417#issuecomment-150096989
@cloud-fan OK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7417#discussion_r41956822
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -274,12 +275,30 @@ private[sql] abstract class
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7931#discussion_r41853694
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
---
@@ -124,6 +124,7 @@ private[hive] class
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7417#discussion_r41855351
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/CartesianProduct.scala
---
@@ -28,9 +28,17 @@ import
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/7853#issuecomment-141082236
@andrewor14 I have set stopped to private[spark], @liancheng @yhuai any
thoughts?
---
If your project is set up for it, you can reply to this email and have your
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/7417#issuecomment-138554397
@scwf done. @zsxwing updated code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7417#discussion_r38504238
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/CartesianProduct.scala
---
@@ -27,16 +27,27 @@ import
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/7846#issuecomment-127815406
@vanzin @srowen Updated, thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/7846#issuecomment-126879883
@srowen We need call interrupt in YarnClientSchedulerBackend.stop(),
details see PR #5305 and PR #3143, so even if we call sc.stop() in the finally
block
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/7417#issuecomment-126880682
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/7853
[SPARK-9522][SQL] SparkSubmit process can not exit if kill application when
HiveThriftServer was starting
When we start HiveThriftServer, we will start SparkContext first, then
start
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/7846#issuecomment-12629
Yes, this change doesn't stop this sequence from happening. As monitor
thread is daemon thread, we don't need call interrupt as sc.stop().
Below I am not very
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/7846
[SPARK-9519][Yarn] Confirm stop sc successfully when application was killed
Currently, when we kill application on Yarn, then will call sc.stop() at
Yarn application state monitor thread
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/7417#issuecomment-126856094
@hvanhovell Good suggestion, thank you, updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/7417#issuecomment-123925858
@hvanhovell I use tpc-ds to test, for below SQL clause:
```
with single_value as (
select 1 tpcds_val from date_dim
)
select sum(ss_quantity
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7417#discussion_r35180395
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastCartesianProduct.scala
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7417#discussion_r34754893
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/CartesianProduct.scala
---
@@ -34,7 +34,15 @@ case class CartesianProduct(left
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/7417
[SPARK-9066][SQL] Improve cartesian performance
see jira https://issues.apache.org/jira/browse/SPARK-9066
You can merge this pull request into a Git repository by running:
$ git pull
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/7417#issuecomment-121588200
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/7209#issuecomment-119504817
@liancheng OK, no problem. Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user Sephiroth-Lin closed the pull request at:
https://github.com/apache/spark/pull/7209
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/7209#issuecomment-119064179
@liancheng I have updated, please help to review, thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/7209#issuecomment-118699916
@liancheng OK, good, thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/7209
[SPARK-8811][SQL] Read array struct data from parquet error
JIRA:https://issues.apache.org/jira/browse/SPARK-8811
For example:
we have a table:
```
t1(c1 string, c2
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/6704#issuecomment-110231290
Close it first as PR #6711 can fix NPE, if we find the root cause of why
the `@VisibleForTesting` annotation causes a NPE in the shell then reopen
Github user Sephiroth-Lin closed the pull request at:
https://github.com/apache/spark/pull/6704
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/6704
Run spark-shell cause NullPointerException
see jira https://issues.apache.org/jira/browse/SPARK-8162
JDK: 1.8.0_40
Hadoop: 2.7.0
You can merge this pull request into a Git repository
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/6704#issuecomment-109965178
@srowen I build the Spark with comman **`mvn -Pyarn -Phadoop-2.4
-Dhadoop.version=2.7.0 -Phive -Phive-thriftserver -Psparkr -DskipTests
package`** and run spark
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-109820635
@srowen @vanzin This PR can cleanup correctly. I just mean without this PR
even if we add KILLED status on ApplicationMaster to check, then it can not
cleanup
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6409#discussion_r31500453
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -91,51 +91,54 @@ private[spark] class Client(
* available
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-108165847
@vanzin I have tested again, and below is the result of final status when
we use yarn to kill the application:
\ | YARN UI | Driver Log | AppMaster
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107399469
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6409#discussion_r31490416
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -825,6 +813,9 @@ private[spark] class Client(
* throw
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6409#discussion_r31397611
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -849,6 +852,27 @@ private[spark] class Client
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5643#discussion_r31304130
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastLeftSemiJoinHash.scala
---
@@ -32,36 +32,59 @@ case class
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-106286738
@tgravescs yes, if yarn do it is better, but now it didn't, so as @vanzin
said may be we can do it when launcher, thank you!
---
If your project is set up
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-105715873
@tgravescs I have tested below:
max retried is defaule, use yarn -kill to kill application when application
start running, run SparkPi with parameter 2
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5887#issuecomment-105516789
ping
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/6409
[SPARK-7705][Yarn] Cleanup of .sparkStaging directory fails if application
is killed
As I have tested, if we cancel or kill the app then the final status may be
undefined, killed
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5887#issuecomment-102756602
@andrewor14 what's your opinion?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5887#issuecomment-101951195
@davies what's your opinion now?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/6114#issuecomment-10184
@scwf @yhuai Done, thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/6114
[SPARK-7595][SQL] Window will cause resolve failed with self join
for example:
table: src(key string, value string)
sql: with v1 as(select key, count(value) over (partition by key
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/6053
[SPARK-7526][SparkR] Specify ip of RBackend, MonitorServer and RRDD Socket
server
These R process only used to communicate with JVM process on local, so
binding to localhost is more
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/6053#issuecomment-101105615
@shivaram Yes, I also think there should be no problems, as it is not
system dependent. I will test this on Windows, thank you!
---
If your project is set up
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/6047
[Minor][PySpark] Set PYTHONPATH to python/lib/pyspark.zip rather than
python/pyspark
As PR#5580 we have create pyspark.zip on building and set PYTHONPATH to
python/lib/pyspark.zip, so
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5887#discussion_r29729314
--- Diff: python/pyspark/shuffle.py ---
@@ -362,7 +362,9 @@ def _spill(self):
self.spills += 1
gc.collect() # release
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/5887
[SPARK-7339][PySpark] PySpark shuffle spill memory sometimes are not correct
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Sephiroth
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5887#issuecomment-98901992
Jenkins retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5580#issuecomment-97346388
If user don't use make-distribution.sh and just compile Spark use maven or
sbt, then don't have pyspark.zip. So we really don't need to do the zip in the
code
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5478#issuecomment-96867560
@tgravescs yes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5478#issuecomment-96145643
@andrewor14 @sryza how about your opinions? thanks. @lianhuiwang please
help me review this, thanks.
---
If your project is set up for it, you can reply
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/5684
[PySpark][Minor] Update sql example, so that can read file correctly
To run Spark, default will read file from HDFS if we don't set the schema.
You can merge this pull request into a Git
Github user Sephiroth-Lin closed the pull request at:
https://github.com/apache/spark/pull/5490
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5478#issuecomment-95102969
@andrewor14 Sorry, these days I am busy, now I have update the code. ^-^
---
If your project is set up for it, you can reply to this email and have your
reply
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5478#issuecomment-94331295
@lianhuiwang OK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5256#issuecomment-93915251
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5478#issuecomment-93705830
@andrewor14 @sryza Done, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5478#issuecomment-93724717
@andrewor14 @sryza @WangTaoTheTonic As I have test again, if we install
Spark on each node, then we can set
spark.executorEnv.PYTHONPATH=${SPARK_HOME}/python
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5256#issuecomment-93650104
@srowen OK, thanks.
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5478#issuecomment-93270239
@andrewor14 @sryza Yes, to assume that the python files will already be
present on the slave machines is not very reasonable. But if user want to use
PySpark
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5479#discussion_r28231958
--- Diff:
yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
---
@@ -128,10 +128,14 @@ private[spark] class
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/5490
[SPARK-5689][Doc] Document what can be run in different YARN modes
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Sephiroth-Lin/spark
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5479#discussion_r28210698
--- Diff:
yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
---
@@ -128,10 +128,14 @@ private[spark] class
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/5478
[SPARK-6869][PySpark] Pass PYTHONPATH to executor, so that executor can
read pyspark file from local file system on executor node
From SPARK-1920 and SPARK-1520 we know PySpark on Yarn can
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/5479
[SPARK-6870][Yarn] Catch InterruptedException when yarn application state
monitor thread been interrupted
On PR #5305 we interrupt the monitor thread but forget to catch
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5305#discussion_r27939765
--- Diff:
yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
---
@@ -127,23 +127,11 @@ private[spark] class
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5305#discussion_r2372
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -559,50 +560,56 @@ private[spark] class Client(
var lastState
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5305#issuecomment-88838919
@srowen unit tests failed at run Python app on yarn-cluster mode, I think
this didn't cause by this PR, please ask jenkins to retest, thank you.
---
If your
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5292#discussion_r27636093
--- Diff:
yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
---
@@ -125,6 +125,7 @@ private[spark] class
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5292#discussion_r27642711
--- Diff:
yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
---
@@ -125,6 +125,7 @@ private[spark] class
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5292#discussion_r27647902
--- Diff:
yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
---
@@ -125,6 +125,7 @@ private[spark] class
Github user Sephiroth-Lin closed the pull request at:
https://github.com/apache/spark/pull/5292
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/5305
[SPARK-4346][SPARK-3596][YARN] Commonize the monitor logic
1. YarnClientSchedulerBack.asyncMonitorApplication use
Client.monitorApplication so that commonize the monitor logic
2. Support
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88708773
@tgravescs @srowen @sryza As i have retest again, if we don't populate
hadoop classpath, then in all case it dosen't work. This PR cann't solve this
issue, i
Github user Sephiroth-Lin closed the pull request at:
https://github.com/apache/spark/pull/5294
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5305#issuecomment-88752700
Jenkins, retest please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/5294
[SPARK-1502][YARN]Add config option to not include yarn/mapred cluster
classpath
You can merge this pull request into a Git repository by running:
$ git pull https://github.com
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/5292
[SPARK-3596][YARN]Support changing the yarn client monitor interval
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Sephiroth-Lin/spark
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5292#discussion_r27540657
--- Diff:
yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
---
@@ -125,6 +125,7 @@ private[spark] class
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/5256
Specify ip of python server scoket
In driver now will start a server socket and use a wildcard ip, use
127.0.0.0 is more reasonable, as we only use it by local Python process.
/cc @davies
Github user Sephiroth-Lin closed the pull request at:
https://github.com/apache/spark/pull/4620
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/4620#issuecomment-76895189
@srowen ok, pls help to close this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4747#discussion_r25322767
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -728,6 +746,11 @@ private[spark] object Utils extends Logging
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/4620#issuecomment-75919413
@srowen as PR #4747 will cache the local root directories, then we can
close this PR first. For PR #4747 I think we also need to remove the local
root directories
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/4620#issuecomment-74864163
@srowen ok, thank you. If this subdirectory is really needed, may be we can
add code to delete this subdirectory after jvm exit or sc.stop().
---
If your project
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/4620#issuecomment-74860104
@srowen as in function getOrCreateLocalRootDirs will create a
subdirectory for root local dir, then if we call getLocalDir will create a
subdirectory for root
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/4620#issuecomment-74604408
@srowen yes, this is same as SPARK-5801. In standalone, worker will create
temp directories for executor, so if we create an unnecessary directory for
local root
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/4620
[SPARK-5830][Core]Don't create unnecessary directory for local root dir
Now will create an unnecessary directory for local root directory, and this
directory will not be deleted after
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/4412#issuecomment-73682144
@srowen thank you, please help to check again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4412#discussion_r24404265
--- Diff: core/src/main/scala/org/apache/spark/HttpFileServer.scala ---
@@ -50,6 +50,15 @@ private[spark] class HttpFileServer(
def stop
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/4412#issuecomment-73655978
@srowen thank you, now I add a member to store the reference of the tmp dir
if it was created, please help to check again.
---
If your project is set up
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4412#discussion_r24285284
--- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
@@ -93,6 +93,14 @@ class SparkEnv (
// actorSystem.awaitTermination
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4412#discussion_r24305832
--- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
@@ -93,6 +93,19 @@ class SparkEnv (
// actorSystem.awaitTermination
1 - 100 of 103 matches
Mail list logo
