Github user tigerquoll commented on the issue:
https://github.com/apache/spark/pull/21308
@rdblue when you say "you don't think the API proposed here needs to
support a first-class partition concept", are you referring to the
"DeleteSupport" Interface, or to Dat
Github user tigerquoll commented on the issue:
https://github.com/apache/spark/pull/21308
@rdblue I think our debate is whether we should expose an API to represent
direct operations on partitions in the new datasource api
Github user tigerquoll commented on the issue:
https://github.com/apache/spark/pull/21308
@rdblue Actually: https://issues.apache.org/jira/browse/SPARK-22389.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user tigerquoll commented on the issue:
https://github.com/apache/spark/pull/21308
@rdblue what about those data sources that support record deletion and
partition dropping as two semantically different operations - Kudu and Hbase
being two examples.
All systems
Github user tigerquoll commented on the issue:
https://github.com/apache/spark/pull/21306
So Kudu range partitions support arbitrary sized partition intervals, like
the example below, where the first and last range partition are six months in
size, but the middle partition is one
Github user tigerquoll commented on the issue:
https://github.com/apache/spark/pull/21306
Sure,
I am looking at the point of view of supporting Kudu. Check out
https://kudu.apache.org/docs/schema_design.html#partitioning for some of the
details. In particular
https
Github user tigerquoll commented on the issue:
https://github.com/apache/spark/pull/21308
I am assuming this API was intended to support the "drop partition"
use-case. I'm arguing that adding and deleting partitions deal with a concept
that is a slightly higher concept
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/21306#discussion_r214808418
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/catalog/v2/TableChange.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/21306#discussion_r214806854
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/catalog/v2/TableChange.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache
Github user tigerquoll commented on the issue:
https://github.com/apache/spark/pull/21306
Can we support column range partition predicates please?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user tigerquoll commented on the pull request:
https://github.com/apache/spark/pull/5250#issuecomment-116651383
Correct.
Cheers,Dale.
Date: Fri, 19 Jun 2015 06:01:26 -0700
From: notificati...@github.com
To: sp...@noreply.github.com
CC: tigerqu
Github user tigerquoll closed the pull request at:
https://github.com/apache/spark/pull/5250
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user tigerquoll closed the pull request at:
https://github.com/apache/spark/pull/4937
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user tigerquoll commented on the pull request:
https://github.com/apache/spark/pull/4937#issuecomment-98918915
Ok, no problems.
I can understand why.
Regards,
Dale.
Sent from my iPad
On 4 May 2015, at 9:48 pm, Sean Owen notificati
Github user tigerquoll commented on the pull request:
https://github.com/apache/spark/pull/5368#issuecomment-93705438
Sounds like a reasonable idea, I'll put another PR together.
Regards,Dale.
Date: Wed, 15 Apr 2015 10:25:26 -0700
From: notificati...@github.com
Github user tigerquoll commented on the pull request:
https://github.com/apache/spark/pull/5368#issuecomment-89942497
The name of the RDD is a placeholder. I'm happy to accept any suggestions
you have for a name that makes more sense to you. The InputFormat based
solution you
GitHub user tigerquoll opened a pull request:
https://github.com/apache/spark/pull/5368
[CORE] [SPARK-6593] Provide a HadoopRDD variant that wraps all reads in a
Try
@rxin @sryza
As per attached jira ticket, I was proposing some way of making hadoop IO
more robust
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/5250#discussion_r27377569
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -246,6 +249,15 @@ class HadoopRDD[K, V](
} catch
Github user tigerquoll commented on the pull request:
https://github.com/apache/spark/pull/5250#issuecomment-87681300
If a user can write scala codes that appropriately deals with the problem,
why can't they write spark code to deal with it in parallel? Isn't this what
spark is about
Github user tigerquoll commented on the pull request:
https://github.com/apache/spark/pull/5250#issuecomment-87650287
Hi Sean,
Thanks for your input - your views have helped me refine my thinking on the
matter. I believe that If you take a purist's point of view then yes you can
Github user tigerquoll commented on the pull request:
https://github.com/apache/spark/pull/5250#issuecomment-87822606
To sum up the debate so far there doesn't appear to be any major concerns
about leaving the system in an inconsistent state. The major concern seems to
be about
Github user tigerquoll commented on the pull request:
https://github.com/apache/spark/pull/5250#issuecomment-87825625
I'd Also like to point out the converse of my second paragraph above - we
are also giving a means of converting deterministic application crashing to
deterministic
GitHub user tigerquoll opened a pull request:
https://github.com/apache/spark/pull/5250
[CORE] [SPARK-6593] Provide option for HadoopRDD to skip bad data splits
When reading a large amount of files from HDFS eg. with
sc.textFile(hdfs:///user/cloudera/logs*.gz). If a single split
GitHub user tigerquoll opened a pull request:
https://github.com/apache/spark/pull/4937
[Spark-6214][CORE] - A simple expression language to specify configuration
options
This is a proposal to allow for configuration options to be specified via a
simple expression language
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/3809#discussion_r22334486
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -329,8 +329,11 @@ class SparkContext(config: SparkConf) extends Logging
GitHub user tigerquoll opened a pull request:
https://github.com/apache/spark/pull/3809
spark-core - [SPARK-4787] - Stop sparkcontext properly if a DAGScheduler
init error occurs
[SPARK-4787] Stop SparkContext properly if an exception occurs during
DAGscheduler initialization
Github user tigerquoll commented on the pull request:
https://github.com/apache/spark/pull/3426#issuecomment-68165724
Heh @JoshRosen @sryza , should this patch include a serialVersionUID
attribute on the classes to be serialized to make sure compiler quirks don't
cause different
Github user tigerquoll commented on the pull request:
https://github.com/apache/spark/pull/3426#issuecomment-68166415
http://stackoverflow.com/questions/285793/what-is-a-serialversionuid-and-why-should-i-use-it
seems to be a good summary of the pros and cons of this approach
Github user tigerquoll commented on the pull request:
https://github.com/apache/spark/pull/2516#issuecomment-66909268
No probs, it was actually a nice way of starting to poke through the code
to figure out how things are put together. I'll stick to smaller jobs from now
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r19793754
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -83,216 +79,163 @@ object SparkSubmit {
* (4) the main
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r19793870
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -33,30 +34,25 @@ import org.apache.spark.util.Utils
* a layer over
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r19794246
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -83,216 +79,163 @@ object SparkSubmit {
* (4) the main
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r19794638
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -17,201 +17,286 @@
package org.apache.spark.deploy
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r19795115
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -17,201 +17,286 @@
package org.apache.spark.deploy
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r19795245
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -227,91 +312,92 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on the pull request:
https://github.com/apache/spark/pull/2516#issuecomment-58943431
Interesting, can you give any references as to what single slash file uris
mean? neither RFC 1738 or 1630 seem to mention them, and
http://en.wikipedia.org/wiki
Github user tigerquoll commented on the pull request:
https://github.com/apache/spark/pull/2516#issuecomment-58775682
Hi @vanzin, I've implemented your suggestions, tidied up the code more, and
also added more unit tests to flesh out the test coverage.
---
If your project is set up
Github user tigerquoll commented on the pull request:
https://github.com/apache/spark/pull/2516#issuecomment-58775741
@andrewor14 we could be stepping on each other's toes soon. Have a query
about your work on Utils.ResolveUris. I notice that you produce and test for
file URIs
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18745897
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1479,6 +1479,14 @@ private[spark] object Utils extends Logging
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18209006
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -17,155 +17,195 @@
package org.apache.spark.deploy
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18148421
--- Diff:
core/src/main/resources/org/apache/spark/deploy/spark-submit-defaults.prop ---
@@ -0,0 +1,18 @@
+
+spark.master = local[*]
--- End
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18148846
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +412,173 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18148885
--- Diff:
core/src/test/scala/org/apache/spark/deploy/MergablePropertiesTest.scala ---
@@ -0,0 +1,55 @@
+package org.apache.spark.deploy
--- End
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18190759
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +412,173 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18191705
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +412,173 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18128789
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +412,173 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18128856
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +412,173 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18128941
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +412,173 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18133582
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -17,155 +17,195 @@
package org.apache.spark.deploy
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18133601
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +412,173 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18134461
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +412,173 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18126499
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/ExecutorData.scala ---
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18126510
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/ExecutorData.scala ---
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18126545
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -104,13 +96,15 @@ class
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18126658
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/ExecutorData.scala ---
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18126663
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -126,8 +120,8 @@ class
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18126665
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -85,16 +79,14 @@ class
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18126667
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -297,6 +291,7 @@ class
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18126669
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -179,25 +176,22 @@ class
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18126677
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -149,13 +144,15 @@ class
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18126678
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -149,13 +144,15 @@ class
Github user tigerquoll commented on the pull request:
https://github.com/apache/spark/pull/2516#issuecomment-56777816
vanzin - great feedback. thanks for the effort of going through the code.
I've implemented all the requested changes.
---
If your project is set up
GitHub user tigerquoll opened a pull request:
https://github.com/apache/spark/pull/2533
SPARK-CORE [SPARK-3651] Group common CoarseGrainedSchedulerBackend
variables together
from [SPARK-3651]
In CoarseGrainedSchedulerBackend, we have:
private val executorActor
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18065148
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -62,15 +62,9 @@ class
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18065365
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -85,16 +79,18 @@ class
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18065400
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -104,13 +100,15 @@ class
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18065662
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -126,8 +124,8 @@ class
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18065891
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -149,13 +147,14 @@ class
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18066183
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -179,25 +178,22 @@ class
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2533#discussion_r18066186
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -179,25 +178,22 @@ class
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18066533
--- Diff:
core/src/main/resources/org/apache/spark/deploy/spark-submit-defaults.prop ---
@@ -0,0 +1,18 @@
+
+spark.master = local[*]
--- End
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18066741
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -330,13 +330,13 @@ private[spark] object SparkConf {
*/
def
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18067374
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +413,166 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18067408
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -57,6 +57,10 @@ object SparkSubmit {
private val
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18067608
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -288,11 +291,11 @@ object SparkSubmit {
}
private def
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18067705
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -17,155 +17,195 @@
package org.apache.spark.deploy
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18067718
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -17,155 +17,195 @@
package org.apache.spark.deploy
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18067811
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -17,155 +17,195 @@
package org.apache.spark.deploy
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18067818
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -17,155 +17,195 @@
package org.apache.spark.deploy
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18068019
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -17,155 +17,195 @@
package org.apache.spark.deploy
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18068346
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -17,155 +17,195 @@
package org.apache.spark.deploy
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18068465
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -17,155 +17,195 @@
package org.apache.spark.deploy
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18068498
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -17,155 +17,195 @@
package org.apache.spark.deploy
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18068639
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -188,41 +228,16 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18068723
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +412,173 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18068816
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +412,173 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18068897
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +412,173 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18068968
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +412,173 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18069354
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +412,173 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18070264
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +412,173 @@ private[spark] class SparkSubmitArguments(args
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18070363
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -406,22 +412,173 @@ private[spark] class SparkSubmitArguments(args
GitHub user tigerquoll opened a pull request:
https://github.com/apache/spark/pull/2516
Spark Core - [SPARK-3620] - Refactor of SparkSubmit Argument parsing code
Argument processing seems to have gotten a lot of attention lately, so I
thought I might throw my contribution
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18008077
--- Diff:
core/src/main/resources/org/apache/spark/deploy/spark-submit-defaults.prop ---
@@ -0,0 +1,90 @@
+# The master URL for the cluster egspark
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18008488
--- Diff:
core/src/main/resources/org/apache/spark/deploy/spark-submit-defaults.prop ---
@@ -0,0 +1,90 @@
+# The master URL for the cluster egspark
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18008765
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -48,8 +50,10 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18009041
--- Diff:
core/src/main/scala/org/apache/spark/deploy/MergedPropertyMap.scala ---
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18009050
--- Diff:
core/src/main/scala/org/apache/spark/deploy/MergedPropertyMap.scala ---
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18009125
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -83,7 +88,7 @@ object SparkSubmit {
* (4) the main class
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18009140
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -130,7 +135,7 @@ object SparkSubmit {
if (!Utils.classIsLoadable
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18009318
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -288,11 +291,11 @@ object SparkSubmit {
}
private def
1 - 100 of 128 matches
Mail list logo
