[GitHub] spark pull request: [MINOR][SQL] Replace DataFrameWriter.stream() ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11629


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR][SQL] Replace DataFrameWriter.stream() ...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/11629#issuecomment-194720804
  
Merging in master!



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SPARK-13797] [SQL] Remove proje...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-194720152
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SPARK-13797] [SQL] Remove proje...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-194720154
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52818/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SPARK-13797] [SQL] Remove proje...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-194720120
  
**[Test build #52818 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52818/consoleFull)**
 for PR 11565 at commit 
[`6cf6f44`](https://github.com/apache/spark/commit/6cf6f444697c0bc32dbc09906fe144563a7d66df).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...

[kiszk]

Github user kiszk commented on the pull request:

https://github.com/apache/spark/pull/11301#issuecomment-194719424
  
@Sarutak, I think that join and sort may generate Java code by own 
routines. Can I address other operations in other future PRs incrementally?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SPARK-13797] [SQL] Remove proje...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-194718830
  
**[Test build #52820 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52820/consoleFull)**
 for PR 11565 at commit 
[`44326f1`](https://github.com/apache/spark/commit/44326f11dab33d686885b7f038a1034c1723338e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13577] [yarn] Allow Spark jar to be mul...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11500#issuecomment-194718630
  
Mostly eyeballing it here, but it looks reasonable to me given its goal. 
You'll know a lot more about what needs to happen to make this work than I 
would.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SPARK-13797] [SQL] Remove proje...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/11565#discussion_r55643000
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -384,9 +374,25 @@ object ColumnPruning extends Rule[LogicalPlan] {
 // Eliminate no-op Projects
 case p @ Project(projectList, child) if sameOutput(child.output, 
p.output) => child
 
+// Eliminate no-op Window
+case w: Window if sameOutput(w.child.output, w.output) => w.child
+
+// Convert Aggregate to Project if no aggregate function exists
+case a: Aggregate if !containsAggregates(a.expressions) =>
+  Project(a.aggregateExpressions, a.child)
--- End diff --

Anyway, I think this is a rare case. Will not do it in this PR


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13577] [yarn] Allow Spark jar to be mul...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11500#discussion_r55643032
  
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -653,7 +653,9 @@ private[spark] object SparkConf extends Logging {
 "spark.memory.offHeap.enabled" -> Seq(
   AlternateConfig("spark.unsafe.offHeap", "1.6")),
 "spark.rpc.message.maxSize" -> Seq(
-  AlternateConfig("spark.akka.frameSize", "1.6"))
+  AlternateConfig("spark.akka.frameSize", "1.6")),
+"spark.yarn.jars" -> Seq(
+  AlternateConfig("spark.yarn.jar", "2.0"))
--- End diff --

I'm probably wrong about this, but is it `.archive` that is the new name 
for `.jar` rather than `.jars`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13789] Infer additional constraints fro...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11618#issuecomment-194717639
  
**[Test build #52819 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52819/consoleFull)**
 for PR 11618 at commit 
[`c57db5b`](https://github.com/apache/spark/commit/c57db5b4a3950ba5e653ff3b8cdfa33e157305ef).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SPARK-13797] [SQL] Remove proje...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/11565#discussion_r55642840
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -384,9 +374,25 @@ object ColumnPruning extends Rule[LogicalPlan] {
 // Eliminate no-op Projects
 case p @ Project(projectList, child) if sameOutput(child.output, 
p.output) => child
 
+// Eliminate no-op Window
+case w: Window if sameOutput(w.child.output, w.output) => w.child
+
+// Convert Aggregate to Project if no aggregate function exists
+case a: Aggregate if !containsAggregates(a.expressions) =>
+  Project(a.aggregateExpressions, a.child)
--- End diff --

It only makes sense when the grouping and aggregateExpressions are exactly 
identical. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13578] [core] Modify launch scripts to ...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11591#issuecomment-194716774
  
Seems OK to me, fWIW


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13789] Infer additional constraints fro...

[sameeragarwal]

Github user sameeragarwal commented on a diff in the pull request:

https://github.com/apache/spark/pull/11618#discussion_r55642742
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala 
---
@@ -62,6 +63,25 @@ abstract class QueryPlan[PlanType <: 
QueryPlan[PlanType]] extends TreeNode[PlanT
   }
 
   /**
+   * Infers an additional set of constraints from a given set of equality 
constraints.
+   * For e.g., if an operator has constraints of the form (`a = 5`, `a = 
b`), this returns an
+   * additional constraint of the form `b = 5`
+   */
+  private def inferAdditionalConstraints(constraints: Set[Expression]): 
Set[Expression] = {
+constraints.map {
+  case eq @ EqualTo(l: Attribute, r: Attribute) =>
+(constraints -- Set(eq)).map(_ transform {
+  case a: Attribute if a.semanticEquals(l) => r
+}).union(
+  (constraints -- Set(eq)).map(_ transform {
+case a: Attribute if a.semanticEquals(r) => l
+  }))
+  case _ =>
+Set.empty[Expression]
+}.foldLeft(Set.empty[Expression])(_ union _) -- constraints
--- End diff --

Thank you, fixed!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR][SQL] Replace DataFrameWriter.stream() ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11629#issuecomment-194715899
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52817/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR][SQL] Replace DataFrameWriter.stream() ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11629#issuecomment-194715896
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11178#discussion_r55642612
  
--- Diff: 
launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java ---
@@ -146,10 +146,38 @@ void addOptionString(List cmd, String 
options) {
 boolean isTesting = "1".equals(getenv("SPARK_TESTING"));
 if (prependClasses || isTesting) {
   String scala = getScalaVersion();
-  List projects = Arrays.asList("core", "repl", "mllib", 
"graphx",
-"streaming", "tools", "sql/catalyst", "sql/core", "sql/hive", 
"sql/hive-thriftserver",
-"yarn", "launcher",
-"common/network-common", "common/network-shuffle", 
"common/network-yarn");
+  // All projects except assemblies:
+  List projects = Arrays.asList(
+"common/network-common",
+"common/network-shuffle",
+"common/network-yarn",
+"common/sketch",
+"common/tags",
+"common/unsafe",
+"core",
+"docker-integration-tests",
+"examples",
+"external/akka",
+"external/flume",
+"external/flume-sink",
+"external/kafka",
+"external/mqtt",
+"external/twitter",
+"external/zeromq",
+"extras/kinesis-asl",
--- End diff --

Same here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL] Migrates DataFrame to Datas...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-194715480
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52814/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11178#discussion_r55642583
  
--- Diff: dev/mima ---
@@ -24,24 +24,63 @@ set -e
 FWDIR="$(cd "`dirname "$0"`"/..; pwd)"
 cd "$FWDIR"
 
-echo -e "q\n" | build/sbt oldDeps/update
-rm -f .generated-mima*
+# All default projects except for assemblies:
+SPARK_PROJECTS=(
+  "common/network-common"
+  "common/network-shuffle"
+  "common/network-yarn"
+  "common/sketch"
+  "common/tags"
+  "common/unsafe"
+  "core"
+  "docker-integration-tests"
+  "examples"
+  "external/akka"
+  "external/flume"
+  "external/flume-sink"
+  "external/kafka"
+  "external/mqtt"
+  "external/twitter"
+  "external/zeromq"
+  "extras/kinesis-asl"
--- End diff --

BTW extras/* is now in external/, and docker* directories are also in 
external/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR][SQL] Replace DataFrameWriter.stream() ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11629#issuecomment-194715507
  
**[Test build #52817 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52817/consoleFull)**
 for PR 11629 at commit 
[`70390ae`](https://github.com/apache/spark/commit/70390aecabf8de65756fae9fb5aed441b4be9244).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL] Migrates DataFrame to Datas...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-194715479
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL] Migrates DataFrame to Datas...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-194715305
  
**[Test build #52814 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52814/consoleFull)**
 for PR 11443 at commit 
[`c774023`](https://github.com/apache/spark/commit/c774023d3bfaff0b0c0ce26f49cfad078520f924).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SPARK-13797] [SPARK-13798] [SQL...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/11565#discussion_r55642440
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -384,9 +374,25 @@ object ColumnPruning extends Rule[LogicalPlan] {
 // Eliminate no-op Projects
 case p @ Project(projectList, child) if sameOutput(child.output, 
p.output) => child
 
+// Eliminate no-op Window
+case w: Window if sameOutput(w.child.output, w.output) => w.child
+
+// Convert Aggregate to Project if no aggregate function exists
+case a: Aggregate if !containsAggregates(a.expressions) =>
+  Project(a.aggregateExpressions, a.child)
--- End diff --

I mean, `select a from t group by a` is totally different than `select a 
from t`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SPARK-13797] [SPARK-13798] [SQL...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/11565#discussion_r55642460
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -384,9 +374,25 @@ object ColumnPruning extends Rule[LogicalPlan] {
 // Eliminate no-op Projects
 case p @ Project(projectList, child) if sameOutput(child.output, 
p.output) => child
 
+// Eliminate no-op Window
+case w: Window if sameOutput(w.child.output, w.output) => w.child
+
+// Convert Aggregate to Project if no aggregate function exists
+case a: Aggregate if !containsAggregates(a.expressions) =>
+  Project(a.aggregateExpressions, a.child)
--- End diff --

Yeah, I forgot it... 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SPARK-13797] [SPARK-13798] [SQL...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/11565#discussion_r55642323
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -384,9 +374,25 @@ object ColumnPruning extends Rule[LogicalPlan] {
 // Eliminate no-op Projects
 case p @ Project(projectList, child) if sameOutput(child.output, 
p.output) => child
 
+// Eliminate no-op Window
+case w: Window if sameOutput(w.child.output, w.output) => w.child
+
+// Convert Aggregate to Project if no aggregate function exists
+case a: Aggregate if !containsAggregates(a.expressions) =>
+  Project(a.aggregateExpressions, a.child)
--- End diff --

But `Aggregate` will do grouping, even there is no aggregate expressions, I 
think this is not right.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SPARK-13797] [SPARK-13798] [SQL...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/11565#discussion_r55642338
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -384,9 +374,25 @@ object ColumnPruning extends Rule[LogicalPlan] {
 // Eliminate no-op Projects
 case p @ Project(projectList, child) if sameOutput(child.output, 
p.output) => child
 
+// Eliminate no-op Window
+case w: Window if sameOutput(w.child.output, w.output) => w.child
--- End diff --

yeah, will do


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11615#discussion_r55642314
  
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -270,9 +270,21 @@ private[spark] object JettyUtils extends Logging {
 
   gzipHandlers.foreach(collection.addHandler)
   connectors.foreach(_.setHost(hostName))
+  // As each Acceptor will use one thread, the number of threads 
should at least be the number
+  // of acceptors plus 1.
+  val minThreads =
+connectors.collect { case c: AbstractConnector => c 
}.map(_.getAcceptors).sum + 1
   server.setConnectors(connectors.toArray)
 
   val pool = new QueuedThreadPool
+  var maxThreads = conf.getInt("spark.ui.threads", pool.getMaxThreads)
--- End diff --

Do we even need config here? when would I otherwise want to control this? 
It seems like you want to always make the pool have at least acceptors+1 
threads.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13788][MLLIB] Fix side effects in Chole...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11617#issuecomment-194713989
  
Certainly, the lapack methods modify their arguments in order to return a 
value. That's by design of course. And the javadoc for this method indicates 
that the `solve()` method also intends to modify its args in place. Your tests 
show this is required. This isn't a bug then.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SPARK-13797] [SPARK-13798] [SQL...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/11565#discussion_r55642207
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -384,9 +374,25 @@ object ColumnPruning extends Rule[LogicalPlan] {
 // Eliminate no-op Projects
 case p @ Project(projectList, child) if sameOutput(child.output, 
p.output) => child
 
+// Eliminate no-op Window
+case w: Window if sameOutput(w.child.output, w.output) => w.child
--- End diff --

`w.windowExpressions.empty` looks simplier?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13706] [ML] Add Python Example for Trai...

[MLnick]

Github user MLnick commented on the pull request:

https://github.com/apache/spark/pull/11547#issuecomment-194713380
  
@JeremyNixon thanks! Merged to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SPARK-13797] [SPARK-13798] [SQL...

[gatorsmile]

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-194713298
  
In the latest PR: we added two more cases/rules into the Optimizer. 

- After column pruning, if `Window` could contain an empty 
`windowExpressions` list, we can remove it
- After column pruning, if `Aggregate` does not contain any 
`AggregateExpression`, we can replace it by `Project`. Then, `Project` can be 
collapsed or pushed down further.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13706] [ML] Add Python Example for Trai...

[MLnick]

Github user MLnick commented on a diff in the pull request:

https://github.com/apache/spark/pull/11547#discussion_r55642006
  
--- Diff: examples/src/main/python/ml/train_validation_split.py ---
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark import SparkContext
+# $example on$
+from pyspark.ml import Pipeline
+from pyspark.ml.evaluation import RegressionEvaluator
+from pyspark.ml.regression import LinearRegression
+from pyspark.ml.tuning import ParamGridBuilder, TrainValidationSplit
+from pyspark.sql import SQLContext
+# $example off$
+
+"""
+This example demonstrates applying TrainValidationSplit to split data
+and preform model selection, as well as applying Pipelines.
--- End diff --

I just cleaned up the import and comment for Pipelines, since it's not 
actually used in this example.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13706] [ML] Add Python Example for Trai...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11547


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13746][Tests]stop using deprecated Sync...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11580#issuecomment-194709904
  
This needs a rebase anyway so I left a few more comments. Otherwise LGTM 
pending tests


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13746][Tests]stop using deprecated Sync...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11580#discussion_r55641453
  
--- Diff: core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala ---
@@ -579,17 +579,20 @@ class CleanerTester(
 
   private def uncleanedResourcesToString = {
 s"""
-  |\tRDDs = ${toBeCleanedRDDIds.toSeq.sorted.mkString("[", ", ", "]")}
-  |\tShuffles = ${toBeCleanedShuffleIds.toSeq.sorted.mkString("[", ", 
", "]")}
-  |\tBroadcasts = ${toBeCleanedBroadcstIds.toSeq.sorted.mkString("[", 
", ", "]")}
+   |\tRDDs = ${toBeCleanedRDDIds.synchronized
--- End diff --

At this point it's getting hard to read this in the interpolated string. 
I'd just pull out the calls to `mkString` and save them as `val`s.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13790] Speed up ColumnVector's getDecim...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/11624#issuecomment-194709645
  
cc @davies 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13746][Tests]stop using deprecated Sync...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11580#discussion_r55641420
  
--- Diff: core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala ---
@@ -442,25 +442,25 @@ class CleanerTester(
 checkpointIds: Seq[Long] = Seq.empty)
   extends Logging {
 
-  val toBeCleanedRDDIds = new HashSet[Int] with SynchronizedSet[Int] ++= 
rddIds
-  val toBeCleanedShuffleIds = new HashSet[Int] with SynchronizedSet[Int] 
++= shuffleIds
-  val toBeCleanedBroadcstIds = new HashSet[Long] with 
SynchronizedSet[Long] ++= broadcastIds
-  val toBeCheckpointIds = new HashSet[Long] with SynchronizedSet[Long] ++= 
checkpointIds
+  val toBeCleanedRDDIds = new HashSet[Int] ++= rddIds
--- End diff --

Can this just be `= HashSet(rddIds)` and similarly for the next 4?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SPARK-13797] [SPARK-13798] [SQL...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-194709320
  
**[Test build #52818 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52818/consoleFull)**
 for PR 11565 at commit 
[`6cf6f44`](https://github.com/apache/spark/commit/6cf6f444697c0bc32dbc09906fe144563a7d66df).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13793] [CORE] PipedRDD doesn't propagat...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11628#issuecomment-194703169
  
**[Test build #2626 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2626/consoleFull)**
 for PR 11628 at commit 
[`003b5ac`](https://github.com/apache/spark/commit/003b5ac9eb98272a70345ad6d7499bf224548ba7).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13793] [CORE] PipedRDD doesn't propagat...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11628#issuecomment-194700395
  
**[Test build #2626 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2626/consoleFull)**
 for PR 11628 at commit 
[`003b5ac`](https://github.com/apache/spark/commit/003b5ac9eb98272a70345ad6d7499bf224548ba7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13793] [CORE] PipedRDD doesn't propagat...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/11628#issuecomment-194700039
  
LGTM pending tests.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12343][YARN] Simplify Yarn client and c...

[jerryshao]

Github user jerryshao commented on a diff in the pull request:

https://github.com/apache/spark/pull/11603#discussion_r55639455
  
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -454,10 +478,16 @@ private[spark] class Client(
  *   (3) whether to add these resources to the classpath
  */
 val cachedSecondaryJarLinks = ListBuffer.empty[String]
+val files = sparkConf.get(FILES_TO_DISTRIBUTE).map(p => 
Utils.resolveURIs(p))
+  .orElse(sys.env.get("SPARK_YARN_DIST_FILES"))
+  .orNull
+val archives = sparkConf.get(ARCHIVES_TO_DISTRIBUTE).map(p => 
Utils.resolveURIs(p))
+  .orElse(sys.env.get("SPARK_YARN_DIST_ARCHIVES"))
+  .orNull
 List(
-  (args.addJars, LocalResourceType.FILE, true),
-  (args.files, LocalResourceType.FILE, false),
-  (args.archives, LocalResourceType.ARCHIVE, false)
+  (sparkConf.getOption("spark.jars").orNull, LocalResourceType.FILE, 
true),
--- End diff --

@tgravescs @vanzin I think maybe I already put "spark.jars" into 
distributed cached IIUC. Maybe the problem is that for yarn cluster mode, here 
I both do the distributed cache and SparkContext#addJars things.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13793] [CORE] PipedRDD doesn't propagat...

[tejasapatil]

Github user tejasapatil commented on the pull request:

https://github.com/apache/spark/pull/11628#issuecomment-194694929
  
@rxin : Thanks for the review. I have updated the PR


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10780][ML][WIP] Add initial model to km...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9#issuecomment-194694816
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52816/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13793] [CORE] PipedRDD doesn't propagat...

[tejasapatil]

Github user tejasapatil commented on a diff in the pull request:

https://github.com/apache/spark/pull/11628#discussion_r55639115
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala ---
@@ -133,23 +134,30 @@ private[spark] class PipedRDD[T: ClassTag](
 // Start a thread to feed the process input from our parent's iterator
 new Thread("stdin writer for " + command) {
   override def run() {
-TaskContext.setTaskContext(context)
 val out = new PrintWriter(proc.getOutputStream)
 
-// scalastyle:off println
-// input the pipe context firstly
-if (printPipeContext != null) {
-  printPipeContext(out.println(_))
-}
-for (elem <- firstParent[T].iterator(split, context)) {
-  if (printRDDElement != null) {
-printRDDElement(elem, out.println(_))
-  } else {
-out.println(elem)
+try {
+  TaskContext.setTaskContext(context)
+
+  // scalastyle:off println
+  // input the pipe context firstly
+  if (printPipeContext != null) {
+printPipeContext(out.println(_))
+  }
+  for (elem <- firstParent[T].iterator(split, context)) {
+if (printRDDElement != null) {
+  printRDDElement(elem, out.println(_))
+} else {
+  out.println(elem)
+}
   }
+  // scalastyle:on println
+} catch {
+  case throwable: Throwable =>
--- End diff --

Did this change. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-194694790
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10780][ML][WIP] Add initial model to km...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9#issuecomment-194694815
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-194694793
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52815/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10780][ML][WIP] Add initial model to km...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9#issuecomment-194694692
  
**[Test build #52816 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52816/consoleFull)**
 for PR 9 at commit 
[`31f7b94`](https://github.com/apache/spark/commit/31f7b94caab2790cbbe99091acf4b614e34cf41c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13793] [CORE] PipedRDD doesn't propagat...

[tejasapatil]

Github user tejasapatil commented on a diff in the pull request:

https://github.com/apache/spark/pull/11628#discussion_r55639013
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala ---
@@ -118,6 +118,7 @@ private[spark] class PipedRDD[T: ClassTag](
 
 val proc = pb.start()
 val env = SparkEnv.get
+@volatile var exception: Option[Throwable] = None
--- End diff --

Did this change


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-194694601
  
**[Test build #52815 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52815/consoleFull)**
 for PR 11565 at commit 
[`4dd3e66`](https://github.com/apache/spark/commit/4dd3e66bf5cb098ee2ea3d47f1cb8d59fddf39f6).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11301#issuecomment-194693889
  
**[Test build #52805 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52805/consoleFull)**
 for PR 11301 at commit 
[`a517e90`](https://github.com/apache/spark/commit/a517e9063c4443a61685ef93e1d3dce6be8a953f).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11301#issuecomment-194693946
  
Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11301#issuecomment-194693948
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52805/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13742][Core] Add non-iterator interface...

[viirya]

Github user viirya commented on the pull request:

https://github.com/apache/spark/pull/11578#issuecomment-194692523
  
I made a mistake in previous benchmark for `withReplacement = false`.

Update benchmark here.

Without generated code for sample():

Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
range/sample/sum:   Best/Avg Time(ms)Rate(M/s)   
Per Row(ns)   Relative

---
range/sample/sum codegen=false 16460 / 17161 31.9   
   31.4   1.0X
range/sample/sum codegen=true4081 / 5390128.5   
7.8   4.0X

With generated code for sample():

Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
range/sample/sum:   Best/Avg Time(ms)Rate(M/s)   
Per Row(ns)   Relative

---
range/sample/sum codegen=false 13688 / 17769 38.3   
   26.1   1.0X
range/sample/sum codegen=true3908 / 3970134.2   
7.5   3.5X

The difference is insignificant too.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL] Migrates DataFrame to Datas...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-194691377
  
I talked with Michael - we thought it's fine to throw an exception that 
contains the query plan. That enables the debugging case and we don't need the 
lazy analysis anymore.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL] Migrates DataFrame to Datas...

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-194691016
  
@yhuai `HiveCompatibilitySuite` passed both locally and on Jenkins.

@marmbrus @rxin OK, then I'll merge this one as soon as Jenkins passes. An 
update of the most recent changes and planned follow-ups:

Changes:

- Dataset eager analysis

  At first I tried to fix this issue by passing encoder by name, then 
workaround this issue via lazy vals and customized serialization. But later on 
I felt that this debugging feature probably isn't worth messing up Dataset 
constructor. Since this debugging feature is only used in 
`PlanTest.checkAnswer`, I resorted to `QueryExecution.assertAnalyzed`. Related 
changes are described in the updated PR description. Details can be found in 
[this commit][1].

Follow-ups:

- As Michael commented, I'll remove `collectRows` and `takeRows` and use 
either `collectAsList`/`takeAsList` or casting when appropriate through out the 
code base.
- (Other TODO entries listed in the PR description)

[1]: 
https://github.com/liancheng/spark/commit/bf7a17a68eb0f41a4cf5c926860449beddc0622c


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/11565#discussion_r55637701
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -387,6 +381,14 @@ object ColumnPruning extends Rule[LogicalPlan] {
 // Can't prune the columns on LeafNode
 case p @ Project(_, l: LeafNode) => p
 
+// Prune windowExpressions and child of Window
+case p @ Project(_, w: Window) if (w.outputSet -- 
p.references).nonEmpty =>
+  val newWindowExprs = 
w.windowExpressions.filter(p.references.contains)
+  val newGrandChild = prunedChild(w.child, w.references ++ 
p.references)
--- End diff --

Then, we also can convert `Aggregate` to `Project` if `Aggregate` does not 
have any aggregate function.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13793] [CORE] PipedRDD doesn't propagat...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/11628#issuecomment-194688064
  
Thanks - looking pretty good. Just some minor comments.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13793] [CORE] PipedRDD doesn't propagat...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/11628#discussion_r55637448
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala ---
@@ -133,23 +134,30 @@ private[spark] class PipedRDD[T: ClassTag](
 // Start a thread to feed the process input from our parent's iterator
 new Thread("stdin writer for " + command) {
   override def run() {
-TaskContext.setTaskContext(context)
 val out = new PrintWriter(proc.getOutputStream)
 
-// scalastyle:off println
-// input the pipe context firstly
-if (printPipeContext != null) {
-  printPipeContext(out.println(_))
-}
-for (elem <- firstParent[T].iterator(split, context)) {
-  if (printRDDElement != null) {
-printRDDElement(elem, out.println(_))
-  } else {
-out.println(elem)
+try {
+  TaskContext.setTaskContext(context)
+
+  // scalastyle:off println
+  // input the pipe context firstly
+  if (printPipeContext != null) {
+printPipeContext(out.println(_))
+  }
+  for (elem <- firstParent[T].iterator(split, context)) {
+if (printRDDElement != null) {
+  printRDDElement(elem, out.println(_))
+} else {
+  out.println(elem)
+}
   }
+  // scalastyle:on println
+} catch {
+  case throwable: Throwable =>
--- End diff --

let's catch only NonFatal 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR][SQL] Replace DataFrameWriter.stream() ...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the pull request:

https://github.com/apache/spark/pull/11629#issuecomment-194687743
  
Thank you!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13793] [CORE] PipedRDD doesn't propagat...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/11628#discussion_r55637430
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala ---
@@ -118,6 +118,7 @@ private[spark] class PipedRDD[T: ClassTag](
 
 val proc = pb.start()
 val env = SparkEnv.get
+@volatile var exception: Option[Throwable] = None
--- End diff --

Let's use an AtomicReference here, and we can also get rid of the Option 
(just use null since it is localized scope)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12555][SQL] Add provided test scenario

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/11623#issuecomment-194687095
  
Can you update the description to actually describe the problem? It'd be 
great if it is understandable without having to cross-reference the JIRA ticket.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR][SQL] Replace DataFrameWriter.stream() ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11629#issuecomment-194686722
  
**[Test build #52817 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52817/consoleFull)**
 for PR 11629 at commit 
[`70390ae`](https://github.com/apache/spark/commit/70390aecabf8de65756fae9fb5aed441b4be9244).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR][SQL] Replace DataFrameWriter.stream() ...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/11629#issuecomment-194686733
  
LGTM pending tests.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13742][Core] Add non-iterator interface...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11578#issuecomment-194686319
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13742][Core] Add non-iterator interface...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11578#issuecomment-194686321
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52812/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR][SQL] Replace DataFrameWriter.stream() ...

[dongjoon-hyun]

GitHub user dongjoon-hyun opened a pull request:

https://github.com/apache/spark/pull/11629

[MINOR][SQL] Replace DataFrameWriter.stream() with startStream() in 
comments.

## What changes were proposed in this pull request?

According to #11627 , this PR replace `DataFrameWriter.stream()` with 
`startStream()` in comments of `ContinuousQueryListener.java`.

## How was this patch tested?

Manual. (It changes on comments.)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dongjoon-hyun/spark minor_rename

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11629.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11629


commit 70390aecabf8de65756fae9fb5aed441b4be9244
Author: Dongjoon Hyun 
Date:   2016-03-10T05:54:27Z

Replace DataFrameWriter.stream() with startStream() in comments.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13742][Core] Add non-iterator interface...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11578#issuecomment-194686038
  
**[Test build #52812 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52812/consoleFull)**
 for PR 11578 at commit 
[`4676940`](https://github.com/apache/spark/commit/4676940d0f7928f5f4d5768d5dbe7670fcf3154b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13734][SPARKR] Added histogram function

[felixcheung]

Github user felixcheung commented on the pull request:

https://github.com/apache/spark/pull/11569#issuecomment-194684792
  
I agree that SparkR should not "require" ggplot...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10780][ML][WIP] Add initial model to km...

[yinxusen]

Github user yinxusen commented on the pull request:

https://github.com/apache/spark/pull/9#issuecomment-194684818
  
test it please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10780][ML][WIP] Add initial model to km...

[yinxusen]

Github user yinxusen commented on a diff in the pull request:

https://github.com/apache/spark/pull/9#discussion_r55636801
  
--- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala 
---
@@ -137,6 +138,17 @@ class KMeansModel private[ml] (
   @Since("1.6.0")
   override def write: MLWriter = new KMeansModel.KMeansModelWriter(this)
 
+  override def hashCode(): Int = {
+(Array(this.getClass, uid) ++ clusterCenters)
--- End diff --

@holdenk Is this one looks good?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10780][ML][WIP] Add initial model to km...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9#issuecomment-194684711
  
**[Test build #52816 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52816/consoleFull)**
 for PR 9 at commit 
[`31f7b94`](https://github.com/apache/spark/commit/31f7b94caab2790cbbe99091acf4b614e34cf41c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/11565#discussion_r55636439
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -387,6 +381,14 @@ object ColumnPruning extends Rule[LogicalPlan] {
 // Can't prune the columns on LeafNode
 case p @ Project(_, l: LeafNode) => p
 
+// Prune windowExpressions and child of Window
+case p @ Project(_, w: Window) if (w.outputSet -- 
p.references).nonEmpty =>
+  val newWindowExprs = 
w.windowExpressions.filter(p.references.contains)
+  val newGrandChild = prunedChild(w.child, w.references ++ 
p.references)
--- End diff --

haha, based on your suggestion, I found another rule to eliminate useless 
`Window`. : )


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12718][SPARK-13720][SQL] SQL generation...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11555#issuecomment-194683067
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12718][SPARK-13720][SQL] SQL generation...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11555#issuecomment-194683068
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52813/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12718][SPARK-13720][SQL] SQL generation...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11555#issuecomment-194682902
  
**[Test build #52813 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52813/consoleFull)**
 for PR 11555 at commit 
[`dab7a2f`](https://github.com/apache/spark/commit/dab7a2f1a5cc0438405b0fa1cf532ab883bed7e7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13789] Infer additional constraints fro...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/11618#discussion_r55636107
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala 
---
@@ -62,6 +63,25 @@ abstract class QueryPlan[PlanType <: 
QueryPlan[PlanType]] extends TreeNode[PlanT
   }
 
   /**
+   * Infers an additional set of constraints from a given set of equality 
constraints.
+   * For e.g., if an operator has constraints of the form (`a = 5`, `a = 
b`), this returns an
+   * additional constraint of the form `b = 5`
+   */
+  private def inferAdditionalConstraints(constraints: Set[Expression]): 
Set[Expression] = {
+constraints.map {
+  case eq @ EqualTo(l: Attribute, r: Attribute) =>
+(constraints -- Set(eq)).map(_ transform {
+  case a: Attribute if a.semanticEquals(l) => r
+}).union(
+  (constraints -- Set(eq)).map(_ transform {
+case a: Attribute if a.semanticEquals(r) => l
+  }))
+  case _ =>
+Set.empty[Expression]
+}.foldLeft(Set.empty[Expression])(_ union _) -- constraints
--- End diff --

maybe not foldLeft -- most of the time fold is used it can be rewritten 
into something imperative that is simpler (and a lot faster)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10780][ML][WIP] Add initial model to km...

[yinxusen]

Github user yinxusen commented on a diff in the pull request:

https://github.com/apache/spark/pull/9#discussion_r55636113
  
--- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala 
---
@@ -137,6 +138,18 @@ class KMeansModel private[ml] (
   @Since("1.6.0")
   override def write: MLWriter = new KMeansModel.KMeansModelWriter(this)
 
+  override def hashCode(): Int =
+this.getClass.hashCode() + uid.hashCode() + 
clusterCenters.map(_.hashCode()).sum
+
+  override def equals(other: Any): Boolean = other match {
+case that: KMeansModel =>
+  this.uid == that.uid &&
+this.clusterCenters.length == that.clusterCenters.length &&
--- End diff --

I'll use sameElements for Array.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/11565#discussion_r55636068
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -387,6 +381,14 @@ object ColumnPruning extends Rule[LogicalPlan] {
 // Can't prune the columns on LeafNode
 case p @ Project(_, l: LeafNode) => p
 
+// Prune windowExpressions and child of Window
+case p @ Project(_, w: Window) if (w.outputSet -- 
p.references).nonEmpty =>
+  val newWindowExprs = 
w.windowExpressions.filter(p.references.contains)
+  val newGrandChild = prunedChild(w.child, w.references ++ 
p.references)
--- End diff --

I see your points. Yeah, you are right. We should use the `newWindowExprs`. 

For the second part, we still need `p.references`. After this PR, the 
`output` of `Window` is ```child.output ++ 
windowExpression.map(_.toAttribute)```. Thus, the attributes used in `Project` 
have to be kept. Is my understanding right?

How about this?
```
prunedChild(w.child, AttributeSet(newWindowExprs.flatMap(_.references)) ++ 
p.references)
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11888] [ML] Decision tree persistence i...

[yinxusen]

Github user yinxusen commented on the pull request:

https://github.com/apache/spark/pull/11581#issuecomment-194675931
  
@jkbradley Make a pass. LGTM except for the minor issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12343][YARN] Simplify Yarn client and c...

[jerryshao]

Github user jerryshao commented on a diff in the pull request:

https://github.com/apache/spark/pull/11603#discussion_r55635356
  
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala ---
@@ -177,14 +177,26 @@ package object config {
 
   private[spark] val DRIVER_CORES = ConfigBuilder("spark.driver.cores")
 .intConf
-.optional
+.withDefault(1)
 
   private[spark] val DRIVER_MEMORY_OVERHEAD = 
ConfigBuilder("spark.yarn.driver.memoryOverhead")
 .bytesConf(ByteUnit.MiB)
 .optional
 
+  private[spark] val DRIVER_MEMORY = ConfigBuilder("spark.driver.memory")
--- End diff --

I see, will change it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7420][STREAMING][TESTS] Enable test: o....

[lw-lin]

Github user lw-lin commented on the pull request:

https://github.com/apache/spark/pull/11626#issuecomment-194672903
  
@rxin thanks !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7420][STREAMING][TESTS] Enable test: o....

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11626


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/11565#discussion_r55634864
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -387,6 +381,14 @@ object ColumnPruning extends Rule[LogicalPlan] {
 // Can't prune the columns on LeafNode
 case p @ Project(_, l: LeafNode) => p
 
+// Prune windowExpressions and child of Window
+case p @ Project(_, w: Window) if (w.outputSet -- 
p.references).nonEmpty =>
+  val newWindowExprs = 
w.windowExpressions.filter(p.references.contains)
+  val newGrandChild = prunedChild(w.child, w.references ++ 
p.references)
--- End diff --

`w.references ++ p.references` are not the attributes we need from child 
right? First, we may filter out some window expressions and `w.references` is 
out of date. Second, `w.child.output` will never contains `p.reference`, as 
`Window` operator only produce window function results.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7420][STREAMING][TESTS] Enable test: o....

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/11626#issuecomment-194671489
  
Thanks - merging in master.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13747][SQL] Fix concurrent query with f...

[tedyu]

Github user tedyu commented on a diff in the pull request:

https://github.com/apache/spark/pull/11586#discussion_r55634802
  
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 
---
@@ -613,7 +613,12 @@ class DAGScheduler(
   properties: Properties): Unit = {
 val start = System.nanoTime
 val waiter = submitJob(rdd, func, partitions, callSite, resultHandler, 
properties)
-Await.ready(waiter.completionFuture, atMost = Duration.Inf)
+// Note: Do not call Await.ready(future) because that calls 
`scala.concurrent.blocking`,
--- End diff --

BlockManager#doPut() does this on line 891:
```
Await.ready(replicationFuture, Duration.Inf)
```
Should that be replaced as well ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-194669064
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52790/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-194668856
  
**[Test build #52790 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52790/consoleFull)**
 for PR 11565 at commit 
[`b229ea2`](https://github.com/apache/spark/commit/b229ea2cfb1dfacb78e1a19fbf3897fdb890c977).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-194669060
  
Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13794][SQL] Rename DataFrameWriter.stre...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/11627#issuecomment-194669186
  
Thanks - merging in master.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13794][SQL] Rename DataFrameWriter.stre...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11627


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13636][SQL] Directly consume UnsafeRow ...

[viirya]

Github user viirya commented on the pull request:

https://github.com/apache/spark/pull/11484#issuecomment-194668304
  
@davies Comments are addressed. Please let me know if you have further 
comments. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13746][Tests]stop using deprecated Sync...

[wilson888888888]

Github user wilson8 commented on a diff in the pull request:

https://github.com/apache/spark/pull/11580#discussion_r55634450
  
--- Diff: core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala ---
@@ -586,10 +586,12 @@ class CleanerTester(
   }
 
   private def isAllCleanedUp =
-toBeCleanedRDDIds.isEmpty &&
-toBeCleanedShuffleIds.isEmpty &&
-toBeCleanedBroadcstIds.isEmpty &&
-toBeCheckpointIds.isEmpty
+synchronized {
--- End diff --

@srowen 
Fixed.  Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13742][Core] Add non-iterator interface...

[viirya]

Github user viirya commented on the pull request:

https://github.com/apache/spark/pull/11578#issuecomment-194667443
  
@davies What do your think? Do we want to use generated codes for sample() 
in wholestage codegen Sample operator?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13794][SQL] Rename DataFrameWriter.stre...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11627#issuecomment-194667169
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52811/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13696] Remove BlockStore class & simpli...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11534#issuecomment-194667138
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52787/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13696] Remove BlockStore class & simpli...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11534#issuecomment-194667136
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13794][SQL] Rename DataFrameWriter.stre...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11627#issuecomment-194667168
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13696] Remove BlockStore class & simpli...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11534#issuecomment-19444
  
**[Test build #52787 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52787/consoleFull)**
 for PR 11534 at commit 
[`e3a4c22`](https://github.com/apache/spark/commit/e3a4c223fcfd943266590a479f7e43d78bafabd5).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org