[GitHub] spark pull request #14427: [SPARK-16818] Exchange reuse incorrectly reuses s...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/14427#discussion_r72898521
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategySuite.scala
 ---
@@ -407,6 +407,39 @@ class FileSourceStrategySuite extends QueryTest with 
SharedSQLContext with Predi
 }
   }
 
+  test("[SPARK-16818] partition pruned file scans implement sameResult 
correctly") {
+withTempPath { path =>
+  val tempDir = path.getCanonicalPath
+  spark.range(100)
+.selectExpr("id", "id as b")
+.write
+.partitionBy("id")
+.parquet(tempDir)
+  val df = spark.read.parquet(tempDir)
+  def getPlan(df: DataFrame): SparkPlan = {
+df.queryExecution.executedPlan
+  }
+  assert(getPlan(df.where("id = 2")).sameResult(getPlan(df.where("id = 
2"
+  assert(!getPlan(df.where("id = 2")).sameResult(getPlan(df.where("id 
= 3"
+}
+  }
+
+  test("[SPARK-16818] exchange reuse respects differences in partition 
pruning") {
+spark.conf.set("spark.sql.exchange.reuse", true)
--- End diff --

ah actually just realized we could've improved with by using "withSQLConf" 
-- it makes sure the configs get reset after the test case finishes running.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14427: [SPARK-16818] Exchange reuse incorrectly reuses scans ov...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14427
  
**[Test build #63053 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63053/consoleFull)**
 for PR 14427 at commit 
[`ef60367`](https://github.com/apache/spark/commit/ef60367331fb3097040cfb0849bdc314c8d399ea).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14427: [SPARK-16818] Exchange reuse incorrectly reuses scans ov...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/14427
  
LGTM pending Jenkins.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14425: [SPARK-16818] Exchange reuse incorrectly reuses scans ov...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/14425
  
Done, see https://github.com/apache/spark/pull/14427


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14427: [SPARK-16818] Exchange reuse incorrectly reuses s...

[ericl]

GitHub user ericl opened a pull request:

https://github.com/apache/spark/pull/14427

[SPARK-16818] Exchange reuse incorrectly reuses scans over different sets 
of partitions

https://github.com/apache/spark/pull/14425 rebased for branch-2.0

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ericl/spark spark-16818-br-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14427.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14427


commit ef60367331fb3097040cfb0849bdc314c8d399ea
Author: Eric Liang 
Date:   2016-07-31T05:48:09Z

[SPARK-16818] Exchange reuse incorrectly reuses scans over different sets 
of partitions

This fixes a bug wherethe file scan operator does not take into account 
partition pruning in its implementation of `sameResult()`. As a result, 
executions may be incorrect on self-joins over the same base file relation.

The patch here is minimal, but we should reconsider relying on `metadata` 
for implementing sameResult() in the future, as string representations may not 
be uniquely identifying.

cc rxin

Unit tests.

Author: Eric Liang 

Closes #14425 from ericl/spark-16818.

Conflicts:

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14425: [SPARK-16818] Exchange reuse incorrectly reuses s...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/14425


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14425: [SPARK-16818] Exchange reuse incorrectly reuses scans ov...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/14425
  
@ericl there is a conflict with branch-2.0. Can you create a pull request 
for branch-2.0?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14425: [SPARK-16818] Exchange reuse incorrectly reuses scans ov...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/14425
  
Merging in master/2.0.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14426: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/14426
  
Why creating a new pull request? All the discussions were in the other pull 
request.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14400: [Spark-16791] [SQL] cast struct with timestamp field fai...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14400
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63052/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14400: [Spark-16791] [SQL] cast struct with timestamp field fai...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14400
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14400: [Spark-16791] [SQL] cast struct with timestamp field fai...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14400
  
**[Test build #63052 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63052/consoleFull)**
 for PR 14400 at commit 
[`6e7f69f`](https://github.com/apache/spark/commit/6e7f69f11e02632837e97c428e39ae3de0926e7b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #9860: SPARK-5106 - Add refresh checkbox on the UI

[sylvinus]

Github user sylvinus commented on the issue:

https://github.com/apache/spark/pull/9860
  
Might I suggest we have another look at this, to save users from breaking 
their cmd-R keys? :-)

This JS snippet (which you can paste in the Chrome console to test) solves 
2 of the issues mentioned here:
 - Doesn't refresh if the Spark UI stopped to avoid losing results
 - Doesn't refresh if the page is not currently visible

```
(function() {

function tinyxhr(url,cb,method,post,contenttype){var 
c=url,a=cb,i=method,f=post,b=contenttype;var d,h;try{h=new 
XMLHttpRequest()}catch(g){try{h=new 
ActiveXObject("Msxml2.XMLHTTP")}catch(g){if(console){console.log("tinyxhr: 
XMLHttpRequest not supported")}return 
null}}d=setTimeout(function(){h.abort();a(new Error("tinyxhr: aborted by a 
timeout"),"",h)},1);h.onreadystatechange=function(){if(h.readyState!=4){return}clearTimeout(d);a(h.status!=200?new
 Error("tinyxhr: server respnse status is 
"+h.status):false,h.responseText,h)};h.open(i?i.toUpperCase():"GET",c,true);if(!f){h.send()}else{h.setRequestHeader("Content-type",b?b:"application/x-www-form-urlencoded");h.send(f)}};

function spark_refresh() {
var delay = 2000;
if (document.hidden) return setTimeout(spark_refresh, delay);
tinyxhr(window.location.toString(), function(error, html) {
if (error) return;
document.documentElement.innerHTML = html;
setTimeout(spark_refresh, delay);
}, "GET");
}

spark_refresh();

})();
```

The `delay` could be configured from a dropdown on the top right corner of 
the page, which is relatively standard in such dashboards.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14419
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63051/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14419
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14419
  
**[Test build #63051 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63051/consoleFull)**
 for PR 14419 at commit 
[`5e92e6f`](https://github.com/apache/spark/commit/5e92e6f59f959772fdc276b40939d4830a70512f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java

[sakky11]

Github user sakky11 commented on the issue:

https://github.com/apache/spark/pull/14402
  
I know its trivial but it still avoids duplication of code and make it more 
readable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14400: [Spark-16791] [SQL] cast struct with timestamp field fai...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14400
  
**[Test build #63052 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63052/consoleFull)**
 for PR 14400 at commit 
[`6e7f69f`](https://github.com/apache/spark/commit/6e7f69f11e02632837e97c428e39ae3de0926e7b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14400: [Spark-16791] [SQL] cast struct with timestamp fi...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14400#discussion_r72897142
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
 ---
@@ -727,6 +727,16 @@ class CastSuite extends SparkFunSuite with 
ExpressionEvalHelper {
 }
   }
 
+  test("cast struct with a timestamp field") {
+val originalSchema = new StructType().add("tsField", TimestampType, 
nullable = false)
+// nine out of ten times I'm casting a struct, it's to normalize its 
fields nullability
+val targetSchema = new StructType().add("tsField", TimestampType, 
nullable = true)
+
+val inp = Literal.create(InternalRow(0L), originalSchema)
+val expected = InternalRow(0L)
+checkEvaluation(cast(inp, targetSchema), expected )
--- End diff --

and remove the extra space here too


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14400: [Spark-16791] [SQL] cast struct with timestamp field fai...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/14400
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #11209: [SPARK-13325][SQL] Create a 64-bit hashcode expre...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/11209#discussion_r72897064
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/HashBenchmark.scala ---
@@ -119,11 +146,12 @@ object HashBenchmark {
   .add("map", mapOfInt)
   .add("mapOfMap", MapType(IntegerType, mapOfInt))
 /*
-Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz
+Intel(R) Core(TM) i7-4750HQ CPU @ 2.00GHz
 Hash For map:   Best/Avg Time(ms)Rate(M/s)   
Per Row(ns)   Relative
 
---
-interpreted version  1820 / 1861  0.0  
444347.2   1.0X
-codegen version   205 /  223  0.0  
 49936.5   8.9X
+interpreted version  1612 / 1618  0.0  
393553.4   1.0X
+codegen version   149 /  150  0.0  
 36381.2  10.8X
+codegen version 64-bit144 /  145  0.0  
 35122.1  11.2X
  */
--- End diff --

The interpreted version is actually calling `BaseGenericInternalRow`'s 
`hashCode` function. It is not calling the expected hash function, right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14398: [SPARK-16774][SQL][WIP] Fix use of deprecated timestamp ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14398
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63050/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14398: [SPARK-16774][SQL][WIP] Fix use of deprecated timestamp ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14398
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14422: Add rand(numRows: Int, numCols: Int) functions

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/spark/pull/14422
  
@HyukjinKwon Thank you. 
This is my first time to push request to spark, Sorrry, I will follow the 
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark  later.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14419
  
**[Test build #63051 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63051/consoleFull)**
 for PR 14419 at commit 
[`5e92e6f`](https://github.com/apache/spark/commit/5e92e6f59f959772fdc276b40939d4830a70512f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14406: [SPARK-16778][SQL][TRIVIAL] Fix deprecation warning with...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14406
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63049/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14406: [SPARK-16778][SQL][TRIVIAL] Fix deprecation warning with...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14406
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #11209: [SPARK-13325][SQL] Create a 64-bit hashcode expre...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/11209#discussion_r72896977
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/HashBenchmark.scala ---
@@ -119,11 +146,12 @@ object HashBenchmark {
   .add("map", mapOfInt)
   .add("mapOfMap", MapType(IntegerType, mapOfInt))
 /*
-Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz
+Intel(R) Core(TM) i7-4750HQ CPU @ 2.00GHz
 Hash For map:   Best/Avg Time(ms)Rate(M/s)   
Per Row(ns)   Relative
 
---
-interpreted version  1820 / 1861  0.0  
444347.2   1.0X
-codegen version   205 /  223  0.0  
 49936.5   8.9X
+interpreted version  1612 / 1618  0.0  
393553.4   1.0X
+codegen version   149 /  150  0.0  
 36381.2  10.8X
+codegen version 64-bit144 /  145  0.0  
 35122.1  11.2X
  */
--- End diff --

I am seeing totally different results. 
```
Java HotSpot(TM) 64-Bit Server VM 1.7.0_80-b15 on Mac OS X 10.11.6
Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz

Hash For map:Best/Avg Time(ms)Rate(M/s)   
Per Row(ns)   Relative


interpreted version  0 /0 85.3  
11.7   1.0X
codegen version215 /  231  0.0  
 52443.1   0.0X
codegen version 64-bit 160 /  173  0.0  
 38973.6   0.0X
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14406: [SPARK-16778][SQL][TRIVIAL] Fix deprecation warning with...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14406
  
**[Test build #63049 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63049/consoleFull)**
 for PR 14406 at commit 
[`9858d99`](https://github.com/apache/spark/commit/9858d9923b6857764391d9cdd0c0332c945cb5c1).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14279: [SPARK-16216][SQL] Write Timestamp and Date in ISO 8601 ...

[deanchen]

Github user deanchen commented on the issue:

https://github.com/apache/spark/pull/14279
  
Would be great to get this in too. Currently using a hack where we iterate 
through all the date columns at the end of our run and manually converting the 
values to the string values formatted as -MM-dd before writing to csv/json.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14118: [SPARK-16462][SPARK-16460][SPARK-15144][SQL] Make CSV ca...

[deanchen]

Github user deanchen commented on the issue:

https://github.com/apache/spark/pull/14118
  
Would be great to get a resolution to this. We're running into issues in 
production attempting to parse csv's with nullable dates. Personally prefer 
option b for our use case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14426: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14426
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14426: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14426
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63048/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14426: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14426
  
**[Test build #63048 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63048/consoleFull)**
 for PR 14426 at commit 
[`ee8bb14`](https://github.com/apache/spark/commit/ee8bb1471b610d634893c145eeaada3045bcaccd).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class Hint(name: String, parameters: Seq[String], child: 
LogicalPlan) extends UnaryNode `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14398: [SPARK-16774][SQL][WIP] Fix use of deprecated timestamp ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14398
  
**[Test build #63050 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63050/consoleFull)**
 for PR 14398 at commit 
[`671f7be`](https://github.com/apache/spark/commit/671f7bef7b953e686365a0f8e33d847b5b0d753f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet co...

[holdenk]

Github user holdenk commented on a diff in the pull request:

https://github.com/apache/spark/pull/14419#discussion_r72896264
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCompatibilityTest.scala
 ---
@@ -119,8 +119,19 @@ private[sql] object ParquetCompatibilityTest {
   metadata: Map[String, String],
   recordWriters: (RecordConsumer => Unit)*): Unit = {
 val messageType = MessageTypeParser.parseMessageType(schema)
-val writeSupport = new DirectWriteSupport(messageType, metadata)
-val parquetWriter = new ParquetWriter[RecordConsumer => Unit](new 
Path(path), writeSupport)
+val testWriteSupport = new DirectWriteSupport(messageType, metadata)
--- End diff --

Sure, so the parquetWriter constructors are deprecated now and its been 
replaced with a builder interface. For Avro and others there is a standard 
builder - but for sort "raw" formats you need to implement your own builder. 
This is equivalent to the old constructor we were using - you can see the 
deprecation in https://github.com/apache/parquet-mr/pull/199/files as well as 
how the builder interface ends up calling an equivalent (now protected) 
constructor. Also since our WriteSupport doesn't need to change based on the 
configuration we always return the same writesupport regardless of conf.

If it would be useful I can add some of this as a comment in the sourcecode.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14406: [SPARK-16778][SQL][TRIVIAL] Fix deprecation warning with...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14406
  
**[Test build #63049 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63049/consoleFull)**
 for PR 14406 at commit 
[`9858d99`](https://github.com/apache/spark/commit/9858d9923b6857764391d9cdd0c0332c945cb5c1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14426: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14426
  
**[Test build #63048 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63048/consoleFull)**
 for PR 14426 at commit 
[`ee8bb14`](https://github.com/apache/spark/commit/ee8bb1471b610d634893c145eeaada3045bcaccd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[dongjoon-hyun]

Github user dongjoon-hyun closed the pull request at:

https://github.com/apache/spark/pull/14132


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/14132
  
This PR grows too much. Sometime, scrolling is lack. I close this and open 
a new one https://github.com/apache/spark/pull/14426 .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14426: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[dongjoon-hyun]

GitHub user dongjoon-hyun opened a pull request:

https://github.com/apache/spark/pull/14426

[SPARK-16475][SQL] Broadcast Hint for SQL Queries

## What changes were proposed in this pull request?
This PR aims to achieve the following two goals in Spark SQL.

**1. Generic Hint Syntax**
The generic hints are parsed and transformed into concrete hints by 
`SubstituteHints` of **Analyzer**. The unknown hints are removed, too. For 
example, `Hint("MAPJOIN")` is transformed into `BroadcastJoin` and other hints 
are removed currently.

```sql
SELECT /*+ MAPJOIN(t) */ * FROM t
SELECT /*+ STREAMTABLE(a,b,c) */ * FROM t
SELECT /*+ INDEX(t emp_job_ix) */ * FROM t
```
Unlink Hive, `NEWMAPJOIN(t)` is allowed for accepting new Spark Hints.


**2. Broadcast Hints**
The followings are recognized. Technically, broadcast hints are matched 
`UnresolvedRelation` to support Hive `MetastoreRelation`. The style of 
`database_name.table_name` is not allowed in this PR.
```sql
SELECT /*+ MAPJOIN(t) */ * FROM t JOIN u ON t.id = u.id
SELECT /*+ BROADCAST(u) */ * FROM t JOIN u ON t.id = u.id
SELECT /*+ BROADCASTJOIN(u) */ * FROM t JOIN u ON t.id = u.id
```

**Examples**
```scala
scala> spark.range(10).createOrReplaceTempView("t")
scala> spark.range(10).createOrReplaceTempView("u")

scala> sql("SELECT * FROM t JOIN u ON t.id = u.id").explain
== Physical Plan ==
*SortMergeJoin [id#0L], [id#4L], Inner
:- *Sort [id#0L ASC], false, 0
:  +- Exchange hashpartitioning(id#0L, 200)
: +- *Range (0, 10, splits=8)
+- *Sort [id#4L ASC], false, 0
   +- ReusedExchange [id#4L], Exchange hashpartitioning(id#0L, 200)

scala> sql("SELECT /*+ MAPJOIN(t) */ * FROM t JOIN u ON t.id = 
u.id").explain
== Physical Plan ==
*BroadcastHashJoin [id#0L], [id#4L], Inner, BuildLeft
:- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, 
false]))
:  +- *Range (0, 10, splits=8)
+- *Range (0, 10, splits=8)

scala> sql("SELECT /*+ MAPJOIN(u) */ * FROM t JOIN u ON t.id = 
u.id").explain
== Physical Plan ==
*BroadcastHashJoin [id#0L], [id#4L], Inner, BuildRight
:- *Range (0, 10, splits=8)
+- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, 
false]))
   +- *Range (0, 10, splits=8)

scala> sql("CREATE TABLE hive_t(id INT)")
res5: org.apache.spark.sql.DataFrame = []

scala> sql("CREATE TABLE hive_u(id INT)")
res6: org.apache.spark.sql.DataFrame = []

scala> sql("SELECT /*+ MAPJOIN(hive_u) */ * FROM hive_t JOIN hive_u ON 
hive_t.id = hive_u.id").explain
== Physical Plan ==
*BroadcastHashJoin [id#28], [id#29], Inner, BuildRight
:- *Filter isnotnull(id#28)
:  +- HiveTableScan [id#28], MetastoreRelation default, hive_t
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, 
false] as bigint)))
   +- *Filter isnotnull(id#29)
  +- HiveTableScan [id#29], MetastoreRelation default, hive_u

scala> sql("SELECT * FROM hive_t JOIN hive_u ON hive_t.id = 
hive_u.id").explain
== Physical Plan ==
*SortMergeJoin [id#36], [id#37], Inner
:- *Sort [id#36 ASC], false, 0
:  +- Exchange hashpartitioning(id#36, 200)
: +- *Filter isnotnull(id#36)
:+- HiveTableScan [id#36], MetastoreRelation default, hive_t
+- *Sort [id#37 ASC], false, 0
   +- Exchange hashpartitioning(id#37, 200)
  +- *Filter isnotnull(id#37)
 +- HiveTableScan [id#37], MetastoreRelation default, hive_u
```

## How was this patch tested?

Pass the Jenkins tests with new testcases.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dongjoon-hyun/spark SPARK-16475-HINT

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14426.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14426


commit ee8bb1471b610d634893c145eeaada3045bcaccd
Author: Dongjoon Hyun 
Date:   2016-07-11T08:04:51Z

[SPARK-16475][SQL] Broadcast Hint for SQL Queries




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14425: [SPARK-16818] Exchange reuse incorrectly reuses scans ov...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14425
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63047/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14425: [SPARK-16818] Exchange reuse incorrectly reuses scans ov...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14425
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14425: [SPARK-16818] Exchange reuse incorrectly reuses scans ov...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14425
  
**[Test build #63047 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63047/consoleFull)**
 for PR 14425 at commit 
[`a254540`](https://github.com/apache/spark/commit/a2545408d144c5ea87ce5696341fe52cd2d29d2c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14425: [SPARK-16818] Exchange reuse incorrectly reuses s...

[ericl]

Github user ericl commented on a diff in the pull request:

https://github.com/apache/spark/pull/14425#discussion_r72895545
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
 ---
@@ -130,7 +130,9 @@ private[sql] object FileSourceStrategy extends Strategy 
with Logging {
   createNonBucketedReadRDD(readFile, selectedPartitions, 
fsRelation)
   }
 
+  // These metadata values make scan plans uniquely identifiable for 
equality checking.
   val meta = Map(
--- End diff --

Agreed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14425: [SPARK-16818] Exchange reuse incorrectly reuses scans ov...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/14425
  
Yep, both fail prior to the fix.

On Sat, Jul 30, 2016, 3:32 PM Reynold Xin  wrote:

> LGTM (assuming the test case would fail without the fix)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14425: [SPARK-16818] Exchange reuse incorrectly reuses s...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/14425#discussion_r72895137
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
 ---
@@ -130,7 +130,9 @@ private[sql] object FileSourceStrategy extends Strategy 
with Logging {
   createNonBucketedReadRDD(readFile, selectedPartitions, 
fsRelation)
   }
 
+  // These metadata values make scan plans uniquely identifiable for 
equality checking.
   val meta = Map(
--- End diff --

not related to this pr - i think we should remove the concept of metadata 
entirely from physical plans. It is something added as a hack to propagate the 
following information, which really should just be a named field in those case 
classes.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14425: [SPARK-16818] Exchange reuse incorrectly reuses scans ov...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/14425
  
LGTM (assuming the test case would fail without the fix)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14425: [SPARK-16818] Exchange reuse incorrectly reuses s...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/14425#discussion_r72895124
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategySuite.scala
 ---
@@ -408,6 +408,39 @@ class FileSourceStrategySuite extends QueryTest with 
SharedSQLContext with Predi
 }
   }
 
+  test("[SPARK-16818] partition pruned file scans implement sameResult 
correctly") {
+withTempPath { path =>
+  val tempDir = path.getCanonicalPath
+  spark.range(100)
+.selectExpr("id", "id as b")
+.write
+.partitionBy("id")
+.parquet(tempDir)
+  val df = spark.read.parquet(tempDir)
+  def getPlan(df: DataFrame): SparkPlan = {
+df.queryExecution.executedPlan
+  }
+  assert(getPlan(df.where("id = 2")).sameResult(getPlan(df.where("id = 
2"
--- End diff --

did you verify this would fail without your patch?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14425: [SPARK-16818] Exchange reuse incorrectly reuses scans ov...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14425
  
**[Test build #63047 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63047/consoleFull)**
 for PR 14425 at commit 
[`a254540`](https://github.com/apache/spark/commit/a2545408d144c5ea87ce5696341fe52cd2d29d2c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14425: [SPARK-16818] Exchange reuse incorrectly reuses s...

[ericl]

GitHub user ericl opened a pull request:

https://github.com/apache/spark/pull/14425

[SPARK-16818] Exchange reuse incorrectly reuses scans over different sets 
of partitions

## What changes were proposed in this pull request?

This fixes a bug wherethe file scan operator does not take into account 
partition pruning in its implementation of `sameResult()`. As a result, 
executions may be incorrect on self-joins over the same base file relation.

The patch here is minimal, but maybe we should reconsider using a metadata 
map for implementing sameResult() in the future.

cc @rxin 

## How was this patch tested?

Unit tests.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ericl/spark spark-16818

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14425.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14425


commit e7e545fd8f7455a653c2bcee4e42a0e5249791f9
Author: Eric Liang 
Date:   2016-07-30T22:02:48Z

Sat Jul 30 15:02:48 PDT 2016




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14412: [SPARK-15355] [CORE] [WIP] Proactive block replic...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/14412#discussion_r72894823
  
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala 
---
@@ -37,10 +37,11 @@ import org.apache.spark.util.Utils
 class BlockManagerId private (
 private var executorId_ : String,
 private var host_ : String,
-private var port_ : Int)
+private var port_ : Int,
+private var topologyInfo_ : Option[String])
--- End diff --

we need to document what information this parameter contains (and in what 
format).



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14415: [SPARK-16810] Refactor registerSinks with multiple const...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/14415
  
DId you add a test?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet co...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/14419#discussion_r72894773
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCompatibilityTest.scala
 ---
@@ -119,8 +119,19 @@ private[sql] object ParquetCompatibilityTest {
   metadata: Map[String, String],
   recordWriters: (RecordConsumer => Unit)*): Unit = {
 val messageType = MessageTypeParser.parseMessageType(schema)
-val writeSupport = new DirectWriteSupport(messageType, metadata)
-val parquetWriter = new ParquetWriter[RecordConsumer => Unit](new 
Path(path), writeSupport)
+val testWriteSupport = new DirectWriteSupport(messageType, metadata)
--- End diff --

can you explain what's going on here? This seems like pretty complicated.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14421: [SPARK-16816] Add api to get JavaSparkContext from Spark...

[petermaxlee]

Github user petermaxlee commented on the issue:

https://github.com/apache/spark/pull/14421
  
Isn't this just

```
new JavaSparkContext(session.sparkContext)
```
?

Perhaps we should just update the documentation to say that.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14422: Add rand(numRows: Int, numCols: Int) functions

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/14422
  
I guess it might be ncer if the contributions you make follow 
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark.just 
like other contributions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/14419
  
No, I don't :). I just meant I wanted to say that this fix is reasonable 
and the reason I didn't submit a PR has not been because there is potential 
incompatability. It was because just I was thinking doing this with other ones 
in the future. I like this PR :).

 Sorry for confusing you.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14417: [SPARK-16812] Open up SparkILoop.getAddedJars

[JoshRosen]

Github user JoshRosen commented on the issue:

https://github.com/apache/spark/pull/14417
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java

[JoshRosen]

Github user JoshRosen commented on the issue:

https://github.com/apache/spark/pull/14402
  
This patch is only a one-line reduction of code and touches a file which 
hasn't needed to be modified in ages; is this change really necessary / worth 
spending review time on?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/14402
  
Did you compile this locally before pushing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14406: [SPARK-16778][SQL][TRIVIAL] Fix deprecation warni...

[holdenk]

Github user holdenk commented on a diff in the pull request:

https://github.com/apache/spark/pull/14406#discussion_r72892893
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfSuite.scala ---
@@ -28,7 +28,7 @@ class SQLConfSuite extends QueryTest with 
SharedSQLContext {
   test("propagate from spark conf") {
 // We create a new context here to avoid order dependence with other 
tests that might call
 // clear().
-val newContext = new SQLContext(sparkContext)
+val newContext = new 
SQLContext(SparkSession.builder().sparkContext(sc).getOrCreate())
--- End diff --

oops thought this built locally but I guess I was on the wrong branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/14419
  
@HyukjinKwon do you have a PR for this? If so I can go ahead and close this 
one (and lets maybe sync some on the deprecation stuff we are both cleaning up 
:))


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/14419
  
So yes as @HyukjinKwon says its just in the tests.
I figured this would be ok across the hadoop versions we target because I 
looked at the build deps files and they all listed parquet-hadoop-1.8.1 for the 
different hadoop profiles we build with (`grep -r parquet-hadoop ./dev/deps/`)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14424: Add test:DenseMatrix.rand with no rng

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/spark/pull/14424
  
It add for https://github.com/apache/spark/pull/14422


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14424: Add test:DenseMatrix.rand with no rng

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14424
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14422: Add rand(numRows: Int, numCols: Int) functions

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/spark/pull/14422
  
I add test : https://github.com/apache/spark/pull/14424



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14424: Add test:DenseMatrix.rand with no rng

[xubo245]

GitHub user xubo245 opened a pull request:

https://github.com/apache/spark/pull/14424

Add test:DenseMatrix.rand with no rng

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)


## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)


(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)


Add test:DenseMatrix.rand with no rng

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xubo245/spark patch-3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14424.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14424


commit 2e945479af45fdadd8abb4529173db04226de64e
Author: 徐波 <601450...@qq.com>
Date:   2016-07-30T18:00:01Z

Add test:DenseMatrix.rand with no rng

Add test:DenseMatrix.rand with no rng




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch cache i...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13713
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch cache i...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13713
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63046/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch cache i...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13713
  
**[Test build #63046 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63046/consoleFull)**
 for PR 13713 at commit 
[`bad9d69`](https://github.com/apache/spark/commit/bad9d695fd9d24d5043182c00a1537953ceb7bd4).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging 
with Serializable `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java

[sakky11]

Github user sakky11 commented on the issue:

https://github.com/apache/spark/pull/14402
  
I think the issue should be fixed now.Sorry for trouble.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14363: [SPARK-16731][SQL] use StructType in CatalogTable and re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14363
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14363: [SPARK-16731][SQL] use StructType in CatalogTable and re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14363
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63045/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14363: [SPARK-16731][SQL] use StructType in CatalogTable and re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14363
  
**[Test build #63045 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63045/consoleFull)**
 for PR 14363 at commit 
[`80d2f50`](https://github.com/apache/spark/commit/80d2f505974f6f94549ccbc5d5014ab83e46a99a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14422: Add rand(numRows: Int, numCols: Int) functions

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/14422
  
Doesn't that need a JIRA, test and PR descriptio maybe?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14380: [SPARK-16485][DOC][ML] Remove useless latex in scaladoc ...

[lins05]

Github user lins05 commented on the issue:

https://github.com/apache/spark/pull/14380
  
@srowen I have looked about those files, but for complex equations, (e.g. 
[this 
one](https://github.com/apache/spark/blob/v2.0.0/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala#L778-L782))
 I found it difficult to strip the latex symbols without affecting the 
readability of the javadoc: With the latex symbols there, the users can copy 
the equation and feed it to a latex viewer like [mathjax online 
previewer](https://cdn.mathjax.org/mathjax/latest/test/sample-dynamic.html), 
but it would make it impossible if we remove latex symbols from it.

Meanwhile I find it's possible to use MathJax to render the latex in the 
scaladoc, as long as we add the necessary javascript to load the MathJax in the 
scaladoc javascript template. I played with it a bit and you can see the 
screenshot:

https://cloud.githubusercontent.com/assets/717363/17271812/d9c40b00-56b8-11e6-9c17-affac4706153.png;>

Do you think it's a good idea to do it?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14422: Add rand(numRows: Int, numCols: Int) functions

[xubo245]

Github user xubo245 commented on a diff in the pull request:

https://github.com/apache/spark/pull/14422#discussion_r72890434
  
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala 
---
@@ -497,6 +497,20 @@ object DenseMatrix {
   }
 
   /**
+* Generate a `DenseMatrix` consisting of `i.i.d.` uniform random 
numbers.
+* 
+* @param numRows number of rows of the matrix
+* @param numCols number of columns of the matrix
+* @return DenseMatrix` with size `numRows` x `numCols` and values in 
U(0, 1)
+*/
+  @Since("2.0.0")
+  def rand(numRows: Int, numCols: Int): DenseMatrix = {
+require(numRows.toLong * numCols <= Int.MaxValue,
+  s"$numRows x $numCols dense matrix is too large to allocate")
+new DenseMatrix(numRows, numCols, Array.fill(numRows * numCols)((new 
Random).nextDouble()))
--- End diff --

Can fix RNG, This makes a new RNG for all element :
val rng = new Random()
new DenseMatrix(numRows, numCols, Array.fill(numRows * 
numCols)(rng.nextDouble()))


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14423: Add zeros(size: Int) function

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14423
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14423: Add zeros(size: Int) function

[xubo245]

GitHub user xubo245 opened a pull request:

https://github.com/apache/spark/pull/14423

Add zeros(size: Int) function

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)


## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)


(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)


Generate a `DenseVector` consisting of zeros.
It can replace breeze.linalg.DenseVector#zeros[Double]

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xubo245/spark patch-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14423.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14423


commit b111cdb9236c87af18e1ea773cea72b73fc68561
Author: 徐波 <601450...@qq.com>
Date:   2016-07-30T16:30:47Z

Add zeros(size: Int) function

Generate a `DenseVector` consisting of zeros.
It can replace breeze.linalg.DenseVector#zeros[Double]




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java

[sakky11]

Github user sakky11 commented on the issue:

https://github.com/apache/spark/pull/14402
  
Hey sorry @srowen ,Actually I have fixed the compilation error but missed 
it in adding in a pull request.Sorry.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/14402
  
@sakky11 you have not even addressed the compilation error. This is wasting 
everyone's time. Please _compile_ this locally and fix it before pinging again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14416: [SPARK-16776][STREAMING] Replace deprecated API in Kafka...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14416
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63044/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14416: [SPARK-16776][STREAMING] Replace deprecated API in Kafka...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14416
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14416: [SPARK-16776][STREAMING] Replace deprecated API in Kafka...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14416
  
**[Test build #63044 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63044/consoleFull)**
 for PR 14416 at commit 
[`a1ef311`](https://github.com/apache/spark/commit/a1ef31129faa3c571bfd58c196ce3757eec890b8).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/14419
  
Could  cc @liancheng who has more insight about thie please?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java

[sakky11]

Github user sakky11 commented on the issue:

https://github.com/apache/spark/pull/14402
  
Added the required fix.Can you please start the build ??


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14422: Add rand(numRows: Int, numCols: Int) functions

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/14422#discussion_r72889871
  
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala 
---
@@ -497,6 +497,20 @@ object DenseMatrix {
   }
 
   /**
+* Generate a `DenseMatrix` consisting of `i.i.d.` uniform random 
numbers.
+* 
+* @param numRows number of rows of the matrix
+* @param numCols number of columns of the matrix
+* @return DenseMatrix` with size `numRows` x `numCols` and values in 
U(0, 1)
+*/
+  @Since("2.0.0")
+  def rand(numRows: Int, numCols: Int): DenseMatrix = {
+require(numRows.toLong * numCols <= Int.MaxValue,
+  s"$numRows x $numCols dense matrix is too large to allocate")
+new DenseMatrix(numRows, numCols, Array.fill(numRows * numCols)((new 
Random).nextDouble()))
--- End diff --

This makes a new RNG for every element, which isn't great.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch cache i...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13713
  
**[Test build #63046 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63046/consoleFull)**
 for PR 13713 at commit 
[`bad9d69`](https://github.com/apache/spark/commit/bad9d695fd9d24d5043182c00a1537953ceb7bd4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch cache i...

[drcrallen]

Github user drcrallen commented on the issue:

https://github.com/apache/spark/pull/13713
  
@tnachen this test is a little larger than I originally anticipated. Let me 
see if I can add some unit tests


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch cache i...

[drcrallen]

Github user drcrallen commented on the issue:

https://github.com/apache/spark/pull/13713
  
OoooOOO master updated to 1.0.0

Fixed merge conflicts


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14231: [SPARK-16586] Change the way the exit code of launcher i...

[zasdfgbnm]

Github user zasdfgbnm commented on the issue:

https://github.com/apache/spark/pull/14231
  
Yes this patch looks clearer


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14422: Add rand(numRows: Int, numCols: Int) functions

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14422
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14422: Add rand(numRows: Int, numCols: Int) functions

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/spark/pull/14422
  
we can use it to replacebreeze.linalg.DenseMatrix.rand(numRows: Int, 
numCols: Int)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14422: Add rand(numRows: Int, numCols: Int) functions

[xubo245]

GitHub user xubo245 opened a pull request:

https://github.com/apache/spark/pull/14422

Add rand(numRows: Int, numCols: Int) functions

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)


## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)


(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)


add rand(numRows: Int, numCols: Int) functions to DenseMatrix object,like 
breeze.linalg.DenseMatrix.rand()

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xubo245/spark patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14422.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14422


commit a7a1261f52112a3bca375dd0bed1c1bc0a2e0ed8
Author: 徐波 <601450...@qq.com>
Date:   2016-07-30T15:43:36Z

Add rand(numRows: Int, numCols: Int) functions

add rand(numRows: Int, numCols: Int) functions to DenseMatrix object,like 
breeze.linalg.DenseMatrix.rand()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14363: [SPARK-16731][SQL] use StructType in CatalogTable and re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14363
  
**[Test build #63045 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63045/consoleFull)**
 for PR 14363 at commit 
[`80d2f50`](https://github.com/apache/spark/commit/80d2f505974f6f94549ccbc5d5014ab83e46a99a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14420: [SPARK-14204] [SQL] register driverClass rather than use...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14420
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63043/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14420: [SPARK-14204] [SQL] register driverClass rather than use...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14420
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14400: [Spark-16791] [SQL] cast struct with timestamp fi...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14400#discussion_r72889243
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
 ---
@@ -727,6 +727,16 @@ class CastSuite extends SparkFunSuite with 
ExpressionEvalHelper {
 }
   }
 
+  test("cast struct with a timestamp field") {
+val originalSchema = new StructType().add( "tsField", TimestampType, 
nullable = false )
--- End diff --

please remove the space in `add( xxx )`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14420: [SPARK-14204] [SQL] register driverClass rather than use...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14420
  
**[Test build #63043 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63043/consoleFull)**
 for PR 14420 at commit 
[`c0bd08e`](https://github.com/apache/spark/commit/c0bd08ef36b8b4ed073fcdc2f88927389593e47f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14405: [SPARK-16800][EXAMPLES][ML] Fix Java examples tha...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/14405


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org