date:20170923

[GitHub] spark pull request #19330: Orderable MapType

2017-09-23 Thread jinxing64

GitHub user jinxing64 opened a pull request:

https://github.com/apache/spark/pull/19330

Orderable MapType

## What changes were proposed in this pull request?

We can make MapType orderable, and thus usable in aggregates and joins.

## How was this patch tested?

Unit tests.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jinxing64/spark SPARK-18134-v2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19330.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19330


commit 2e2b98d2fa9b399280cb4b5dbbfa1aadd3ede951
Author: jinxing 
Date:   2017-09-22T08:43:05Z

Orderable MapType




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19321: [SPARK-22100] [SQL] Make percentile_approx support numer...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19321
  
**[Test build #82108 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82108/testReport)**
 for PR 19321 at commit 
[`0d34053`](https://github.com/apache/spark/commit/0d34053f36b9f03bf3d5cf6ca1b3c13b348d24cb).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19318: [SPARK-22096][ML] use aggregateByKeyLocally in feature f...

2017-09-23 Thread WeichenXu123

Github user WeichenXu123 commented on the issue:

https://github.com/apache/spark/pull/19318
  
The `aggregateByKeyLocally` haven't been implemented. Please wait until 
#19317 finished or merge that PR contents into this, otherwise this PR cannot 
use. We should not open a PR which cannot build.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18924: [SPARK-14371] [MLLIB] OnlineLDAOptimizer should n...

2017-09-23 Thread akopich

Github user akopich commented on a diff in the pull request:

https://github.com/apache/spark/pull/18924#discussion_r140630215
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAOptimizer.scala ---
@@ -462,36 +462,55 @@ final class OnlineLDAOptimizer extends LDAOptimizer {
 val expElogbetaBc = batch.sparkContext.broadcast(expElogbeta)
 val alpha = this.alpha.asBreeze
 val gammaShape = this.gammaShape
+val optimizeDocConcentration = this.optimizeDocConcentration
+// We calculate logphat in the same pass as other statistics, but we 
only need
+// it if we are optimizing docConcentration
+val logphatPartOptionBase = () => if (optimizeDocConcentration) 
Some(BDV.zeros[Double](k))
+  else None
 
-val stats: RDD[(BDM[Double], List[BDV[Double]])] = batch.mapPartitions 
{ docs =>
+val stats: RDD[(BDM[Double], Option[BDV[Double]], Int)] = 
batch.mapPartitions { docs =>
--- End diff --

Done.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18924: [SPARK-14371] [MLLIB] OnlineLDAOptimizer should not coll...

2017-09-23 Thread akopich

Github user akopich commented on the issue:

https://github.com/apache/spark/pull/18924
  
@jkbradley, thanks for the comments. Who is supposed to create the followup 
jira? 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19329: [SPARK-22110][SQL][Documentation] Add usage and improve ...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19329
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19332: [SPARK-22093][TESTS] Fixes `assume` in the tests for `Ut...

2017-09-23 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19332
  
Will check other assumes too soon and be back.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19310: [SPARK-18136] Fix SPARK_JARS_DIR for Python pip install ...

2017-09-23 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19310
  
Merged to master, branch-2.2 and branch-2.1.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19310: [SPARK-18136] Fix SPARK_JARS_DIR for Python pip i...

2017-09-23 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19310


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19331: [SPARK-22109][SQL] Resolves type conflicts between strin...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19331
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19331: [SPARK-22109][SQL] Resolves type conflicts between strin...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19331
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82112/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19330: Orderable MapType

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19330
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82107/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19330: Orderable MapType

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19330
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19330: Orderable MapType

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19330
  
**[Test build #82107 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82107/testReport)**
 for PR 19330 at commit 
[`bd50495`](https://github.com/apache/spark/commit/bd504955d733834fb4b63623ab059294bbf91fc9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19329: [SPARK-22110][SQL][Documentation] Add usage and improve ...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19329
  
**[Test build #82109 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82109/testReport)**
 for PR 19329 at commit 
[`0f3307d`](https://github.com/apache/spark/commit/0f3307dfc3dd21b3d643d3d58e9202743f958b23).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19329: [SPARK-22110][SQL][Documentation] Add usage and improve ...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19329
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82109/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19322: [SPARK-22102][SQL] Set ConfVars.METASTOREWAREHOUSE befor...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19322
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82113/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19322: [SPARK-22102][SQL] Set ConfVars.METASTOREWAREHOUSE befor...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19322
  
**[Test build #82113 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82113/testReport)**
 for PR 19322 at commit 
[`cfc31f7`](https://github.com/apache/spark/commit/cfc31f76dc2a13ec2422963615254209a86af4ca).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19322: [SPARK-22102][SQL] Set ConfVars.METASTOREWAREHOUSE befor...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19322
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18924: [SPARK-14371] [MLLIB] OnlineLDAOptimizer should not coll...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18924
  
**[Test build #82111 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82111/testReport)**
 for PR 18924 at commit 
[`1f9e650`](https://github.com/apache/spark/commit/1f9e650baab01c3b248d5b0225dbec1cde0c737d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18924: [SPARK-14371] [MLLIB] OnlineLDAOptimizer should not coll...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18924
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18924: [SPARK-14371] [MLLIB] OnlineLDAOptimizer should not coll...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18924
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82111/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19310: [SPARK-18136] Fix SPARK_JARS_DIR for Python pip install ...

2017-09-23 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19310
  
@jsnowacki, I am merging this assuming you will fix up 
https://issues.apache.org/jira/browse/SPARK-18136?focusedCommentId=16175276=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16175276
 soon in a followup. I also verified this too.

In a quick look, we probably should fix codes around 
https://github.com/apache/spark/blob/aad2125475dcdeb4a0410392b6706511db17bac4/python/setup.py#L126
 I guess


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19331: [SPARK-22109][SQL] Resolves type conflicts betwee...

2017-09-23 Thread HyukjinKwon

GitHub user HyukjinKwon opened a pull request:

https://github.com/apache/spark/pull/19331

[SPARK-22109][SQL] Resolves type conflicts between strings and timestamps 
in partition column

## What changes were proposed in this pull request?

This PR proposes to resolve the type conflicts in strings and timestamps in 
partition column values.
It looks we need to set the timezone as it needs a cast between strings and 
timestamps.

```scala
val df = Seq((1, "2015-01-01 00:00:00"), (2, "2014-01-01 00:00:00"), (3, 
"blah")).toDF("i", "str")
val path = "/tmp/test.parquet"
df.write.format("parquet").partitionBy("str").save(path)
spark.read.parquet(path).show()
```

**Before**

```
java.util.NoSuchElementException: None.get
  at scala.None$.get(Option.scala:347)
  at scala.None$.get(Option.scala:345)
  at 
org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression$class.timeZone(datetimeExpressions.scala:46)
  at 
org.apache.spark.sql.catalyst.expressions.Cast.timeZone$lzycompute(Cast.scala:172)
  at org.apache.spark.sql.catalyst.expressions.Cast.timeZone(Cast.scala:172)
  at 
org.apache.spark.sql.catalyst.expressions.Cast$$anonfun$castToString$3$$anonfun$apply$16.apply(Cast.scala:208)
  at 
org.apache.spark.sql.catalyst.expressions.Cast$$anonfun$castToString$3$$anonfun$apply$16.apply(Cast.scala:208)
  at 
org.apache.spark.sql.catalyst.expressions.Cast.org$apache$spark$sql$catalyst$expressions$Cast$$buildCast(Cast.scala:201)
  at 
org.apache.spark.sql.catalyst.expressions.Cast$$anonfun$castToString$3.apply(Cast.scala:207)
  at 
org.apache.spark.sql.catalyst.expressions.Cast.nullSafeEval(Cast.scala:533)
  at 
org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:331)
  at 
org.apache.spark.sql.execution.datasources.PartitioningUtils$$anonfun$org$apache$spark$sql$execution$datasources$PartitioningUtils$$resolveTypeConflicts$1.apply(PartitioningUtils.scala:481)
  at 
org.apache.spark.sql.execution.datasources.PartitioningUtils$$anonfun$org$apache$spark$sql$execution$datasources$PartitioningUtils$$resolveTypeConflicts$1.apply(PartitioningUtils.scala:480)
  at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
```

**After**

```
+---+---+
|  i|str|
+---+---+
|  2|2014-01-01 00:00:00|
|  1|2015-01-01 00:00:00|
|  3|   blah|
+---+---+
```

## How was this patch tested?

Unit tests added in `ParquetPartitionDiscoverySuite` and manual tests.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HyukjinKwon/spark SPARK-22109

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19331.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19331


commit 946a0d93ef7baf71500e85156e097a3b9a05888c
Author: hyukjinkwon 
Date:   2017-09-23T10:24:34Z

Resolve type conflicts between strings and timestamps in partition column




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19322: [SPARK-22102][SQL] Set ConfVars.METASTOREWAREHOUSE befor...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19322
  
**[Test build #82115 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82115/testReport)**
 for PR 19322 at commit 
[`e195511`](https://github.com/apache/spark/commit/e195511f550d80ebfd974e52b0b9ab0ce1160011).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19222
  
**[Test build #82116 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82116/testReport)**
 for PR 19222 at commit 
[`8ec08ba`](https://github.com/apache/spark/commit/8ec08ba822bca62db0d917b2908dfc509e08e96b).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19331: [SPARK-22109][SQL] Resolves type conflicts between strin...

2017-09-23 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19331
  
cc @ueshin, could you take a look when you have some time please?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19322: [SPARK-22102][SQL] Set ConfVars.METASTOREWAREHOUSE befor...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19322
  
**[Test build #82115 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82115/testReport)**
 for PR 19322 at commit 
[`e195511`](https://github.com/apache/spark/commit/e195511f550d80ebfd974e52b0b9ab0ce1160011).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19322: [SPARK-22102][SQL] Set ConfVars.METASTOREWAREHOUSE befor...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19322
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82115/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19321: [SPARK-22100] [SQL] Make percentile_approx support numer...

2017-09-23 Thread wzhfy

Github user wzhfy commented on the issue:

https://github.com/apache/spark/pull/19321
  
@felixcheung For percentiles, I think the type of results should be the 
same as input data type. In these tests, the type of data is int, so actually 
`30` is more accurate than `30.0`. The previous answer is `30.0` because 
`ApproximatePercentile` only accepts double input type and outputs double 
results before this PR.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18924: [SPARK-14371] [MLLIB] OnlineLDAOptimizer should not coll...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18924
  
**[Test build #82111 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82111/testReport)**
 for PR 18924 at commit 
[`1f9e650`](https://github.com/apache/spark/commit/1f9e650baab01c3b248d5b0225dbec1cde0c737d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19321: [SPARK-22100] [SQL] Make percentile_approx support numer...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19321
  
**[Test build #82108 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82108/testReport)**
 for PR 19321 at commit 
[`0d34053`](https://github.com/apache/spark/commit/0d34053f36b9f03bf3d5cf6ca1b3c13b348d24cb).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19321: [SPARK-22100] [SQL] Make percentile_approx support numer...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19321
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82108/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19321: [SPARK-22100] [SQL] Make percentile_approx support numer...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19321
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19331: [SPARK-22109][SQL] Resolves type conflicts between strin...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19331
  
**[Test build #82112 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82112/testReport)**
 for PR 19331 at commit 
[`946a0d9`](https://github.com/apache/spark/commit/946a0d93ef7baf71500e85156e097a3b9a05888c).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19322: [SPARK-22102][SQL] Set ConfVars.METASTOREWAREHOUSE befor...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19322
  
**[Test build #82113 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82113/testReport)**
 for PR 19322 at commit 
[`cfc31f7`](https://github.com/apache/spark/commit/cfc31f76dc2a13ec2422963615254209a86af4ca).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19310: [SPARK-18136] Fix SPARK_JARS_DIR for Python pip install ...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19310
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82110/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19310: [SPARK-18136] Fix SPARK_JARS_DIR for Python pip install ...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19310
  
**[Test build #82110 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82110/testReport)**
 for PR 19310 at commit 
[`b5f8ae6`](https://github.com/apache/spark/commit/b5f8ae64c25e9d29bbfcd0f94b9588ab5f2e2cd3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19310: [SPARK-18136] Fix SPARK_JARS_DIR for Python pip install ...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19310
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19332: [SPARK-22093][TESTS] Fixes `assume` in the tests for `Ut...

2017-09-23 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19332
  
@vanzin, Could you take a look when you are available please?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19332: [SPARK-22093][TESTS] Fixes `assume` in the tests ...

2017-09-23 Thread HyukjinKwon

GitHub user HyukjinKwon opened a pull request:

https://github.com/apache/spark/pull/19332

[SPARK-22093][TESTS] Fixes `assume` in the tests for `Utils.resolveURIs` 
and `Utils.resolveURI`

## What changes were proposed in this pull request?

This PR proposes to remove `assume` in `Utils.resolveURIs` and replace 
`assume` to `assert` in `Utils.resolveURI` in the test cases in `UtilsSuite`.

It looks `Utils.resolveURIs` supports multiple but also single paths as 
input. So, it looks not meaningful to check if the input has `,`.

For the test for `Utils.resolveURI`, I replaced it to `assert` because it 
looks taking single path and in order to prevent future mistakes when adding 
more tests here. 

## How was this patch tested?

Fixed unit tests.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HyukjinKwon/spark SPARK-22093

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19332.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19332


commit beaedb09d7b8456f82763b21756f0cd46a9b33fd
Author: hyukjinkwon 
Date:   2017-09-23T10:51:22Z

Fixes `assume` in the tests for Utils.resolveURIs and Utils.resolveURI




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19332: [SPARK-22093][TESTS] Fixes `assume` in the tests for `Ut...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19332
  
**[Test build #82114 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82114/testReport)**
 for PR 19332 at commit 
[`beaedb0`](https://github.com/apache/spark/commit/beaedb09d7b8456f82763b21756f0cd46a9b33fd).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18924: [SPARK-14371] [MLLIB] OnlineLDAOptimizer should not coll...

2017-09-23 Thread WeichenXu123

Github user WeichenXu123 commented on the issue:

https://github.com/apache/spark/pull/18924
  
@akopich follow-up JIRA created here
https://issues.apache.org/jira/browse/SPARK-22111
Can you create follow up PR after this PR being merged ?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19332: [SPARK-22093][TESTS] Fixes `assume` in the tests for `Ut...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19332
  
**[Test build #82117 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82117/testReport)**
 for PR 19332 at commit 
[`c32f65b`](https://github.com/apache/spark/commit/c32f65b4a25c53e3fbf3ca5dff274f762da37d5b).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19332: [SPARK-22093][TESTS] Fixes `assume` in the tests for `Ut...

2017-09-23 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19332
  
I checked other instances of `org.scalatest.Assertions.assume`. I am quite 
sure of the ones here I added. I didn't add ones I am less sure of.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19331: [SPARK-22109][SQL] Resolves type conflicts between strin...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19331
  
**[Test build #82112 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82112/testReport)**
 for PR 19331 at commit 
[`946a0d9`](https://github.com/apache/spark/commit/946a0d93ef7baf71500e85156e097a3b9a05888c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19320: [SPARK-22099] The 'job ids' list style needs to be chang...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19320
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19320: [SPARK-22099] The 'job ids' list style needs to be chang...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19320
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82103/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19330: Orderable MapType

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19330
  
**[Test build #82106 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82106/testReport)**
 for PR 19330 at commit 
[`2e2b98d`](https://github.com/apache/spark/commit/2e2b98d2fa9b399280cb4b5dbbfa1aadd3ede951).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19330: Orderable MapType

2017-09-23 Thread jinxing64

Github user jinxing64 commented on a diff in the pull request:

https://github.com/apache/spark/pull/19330#discussion_r140627825
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
 ---
@@ -663,6 +663,18 @@ class CodegenContext {
   }
 """
   s"${addNewFunction(compareFunc, funcCode)}($c1, $c2)"
+case map: MapType =>
+  val compareFunc = freshName("compareMap")
+  val compare = this.addReferenceObj("compare", 
map.interpretedOrdering,
+classOf[Ordering[MapData]].getCanonicalName)
--- End diff --

TODO: we should code generate this.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13599
  
**[Test build #82104 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82104/testReport)**
 for PR 13599 at commit 
[`abdf7b7`](https://github.com/apache/spark/commit/abdf7b7a8a75dfc7b8de597611bbfa0af126e24e).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19330: Orderable MapType

2017-09-23 Thread hvanhovell

Github user hvanhovell commented on the issue:

https://github.com/apache/spark/pull/19330
  
@jinxing64 thanks for taking over. I have glanced over the PR, and I miss 
the explicit sorting of maps. We can't assume that maps are sorted out of the 
box, for example the following example should evaluate to `true` but it won't 
in your PR: `map(1, 2, 3, 4) = map(3, 4, 1, 2)`.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19310: [SPARK-18136] Fix SPARK_JARS_DIR for Python pip install ...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19310
  
**[Test build #82110 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82110/testReport)**
 for PR 19310 at commit 
[`b5f8ae6`](https://github.com/apache/spark/commit/b5f8ae64c25e9d29bbfcd0f94b9588ab5f2e2cd3).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19310: [SPARK-18136] Fix SPARK_JARS_DIR for Python pip install ...

2017-09-23 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/19310
  
@holdenk @ueshin @HyukjinKwon as it says python pip package, but I'm pretty 
this is cross language


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19321: [SPARK-22100] [SQL] Make percentile_approx suppor...

2017-09-23 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/19321#discussion_r140627874
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
 ---
@@ -85,7 +85,8 @@ case class ApproximatePercentile(
   private lazy val accuracy: Int = 
accuracyExpression.eval().asInstanceOf[Int]
 
   override def inputTypes: Seq[AbstractDataType] = {
-Seq(DoubleType, TypeCollection(DoubleType, ArrayType(DoubleType)), 
IntegerType)
+Seq(TypeCollection(NumericType, DateType, TimestampType),
+  TypeCollection(DoubleType, ArrayType(DoubleType)), IntegerType)
--- End diff --

This will cause the result difference. We need to document it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19330: Orderable MapType

2017-09-23 Thread jinxing64

Github user jinxing64 commented on the issue:

https://github.com/apache/spark/pull/19330
  
It seems https://github.com/apache/spark/pull/15970 is not being worked.
I resolved conflicts and add some tests in this pr.



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19330: Orderable MapType

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19330
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19330: Orderable MapType

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19330
  
**[Test build #82106 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82106/testReport)**
 for PR 19330 at commit 
[`2e2b98d`](https://github.com/apache/spark/commit/2e2b98d2fa9b399280cb4b5dbbfa1aadd3ede951).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19329: [SPARK-22110][SQL][Documentation] Add usage and improve ...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19329
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82105/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19330: Orderable MapType

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19330
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82106/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19329: [SPARK-22110][SQL][Documentation] Add usage and improve ...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19329
  
**[Test build #82105 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82105/testReport)**
 for PR 19329 at commit 
[`0f3307d`](https://github.com/apache/spark/commit/0f3307dfc3dd21b3d643d3d58e9202743f958b23).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19320: [SPARK-22099] The 'job ids' list style needs to be chang...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19320
  
**[Test build #82103 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82103/testReport)**
 for PR 19320 at commit 
[`5cb6ea4`](https://github.com/apache/spark/commit/5cb6ea405755fd70ca6f2f7078914cc9dece8c73).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18936: [SPARK-21688][ML][MLLIB] make native BLAS the first choi...

2017-09-23 Thread srowen

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/18936
  
Interesting, I wouldn't have expected much difference at all. Once it's in 
native code these are all just SSE instructions on the silicon... I don't know 
how it could be much different. But naturally I am not an expert on it.

Anyway, the problem here is that you can't selectively use F2J in some 
cases and not others, so it would potentially slow down OpenBLAS users.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19310: [SPARK-18136] Fix SPARK_JARS_DIR for Python pip install ...

2017-09-23 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/19310
  
I'm a little puzzled how this wouldn't broken on windows without this fix 
with release build... need to run some tests to see.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13599
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82104/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19329: [SPARK-22110][SQL][Documentation] Add usage and improve ...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19329
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19330: Orderable MapType

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19330
  
**[Test build #82107 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82107/testReport)**
 for PR 19330 at commit 
[`bd50495`](https://github.com/apache/spark/commit/bd504955d733834fb4b63623ab059294bbf91fc9).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13599
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19329: [SPARK-22110][SQL][Documentation] Add usage and improve ...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19329
  
**[Test build #82109 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82109/testReport)**
 for PR 19329 at commit 
[`0f3307d`](https://github.com/apache/spark/commit/0f3307dfc3dd21b3d643d3d58e9202743f958b23).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19310: [SPARK-18136] Fix SPARK_JARS_DIR for Python pip install ...

2017-09-23 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/19310
  
jenkins, test this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19329: [SPARK-22110][SQL][Documentation] Add usage and improve ...

2017-09-23 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19329
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19332: [SPARK-22093][TESTS] Fixes `assume` in the tests for `Ut...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19332
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19332: [SPARK-22093][TESTS] Fixes `assume` in the tests for `Ut...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19332
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82114/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19277: [SPARK-22058][CORE]the BufferedInputStream will not be c...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19277
  
**[Test build #3933 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3933/testReport)**
 for PR 19277 at commit 
[`2e5f21a`](https://github.com/apache/spark/commit/2e5f21a1a89bbf69431f918d3043ff6b58420dd8).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17743: [SPARK-20448][DOCS] Document how FileInputDStream works ...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17743
  
**[Test build #3934 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3934/testReport)**
 for PR 17743 at commit 
[`1e620ce`](https://github.com/apache/spark/commit/1e620ceb7b5eb0df6df83525366ebc1074f8e8ce).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19331: [SPARK-22109][SQL] Resolves type conflicts betwee...

2017-09-23 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19331


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19331: [SPARK-22109][SQL] Resolves type conflicts between strin...

2017-09-23 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19331
  
Definitely.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19333: [SPARK-22109][SQL][BRANCH-2.2] Resolves type conflicts b...

2017-09-23 Thread ueshin

Github user ueshin commented on the issue:

https://github.com/apache/spark/pull/19333
  
LGTM, pending Jenkins.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17743: [SPARK-20448][DOCS] Document how FileInputDStream...

2017-09-23 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/17743


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest commit sh...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19290
  
**[Test build #82120 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82120/testReport)**
 for PR 19290 at commit 
[`7e6c2c5`](https://github.com/apache/spark/commit/7e6c2c546d86946ee4cd110321ae24226cb50917).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19332: [SPARK-22093][TESTS] Fixes `assume` in `UtilsSuite` and ...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19332
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82117/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19332: [SPARK-22093][TESTS] Fixes `assume` in `UtilsSuite` and ...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19332
  
**[Test build #82117 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82117/testReport)**
 for PR 19332 at commit 
[`c32f65b`](https://github.com/apache/spark/commit/c32f65b4a25c53e3fbf3ca5dff274f762da37d5b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19332: [SPARK-22093][TESTS] Fixes `assume` in `UtilsSuite` and ...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19332
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadi...

2017-09-23 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/19286#discussion_r140638128
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala 
---
@@ -749,6 +749,34 @@ class JDBCSuite extends SparkFunSuite
 assert(agg.isCascadingTruncateTable() === Some(true))
   }
 
+  test("Aggregated dialects: isCascadingTruncateTable") {
+def genDialect(cascadingTruncateTable: Option[Boolean]): JdbcDialect = 
new JdbcDialect {
+  override def canHandle(url: String): Boolean = true
+  override def getCatalystType(
+sqlType: Int,
+typeName: String,
+size: Int,
+md: MetadataBuilder): Option[DataType] = None
+  override def isCascadingTruncateTable(): Option[Boolean] = 
cascadingTruncateTable
+}
+
+val dialectCombination = Seq(
+  List(genDialect(Some(true)), genDialect(Some(false)), 
genDialect(None)),
+  List(genDialect(Some(true)), genDialect(Some(true)), 
genDialect(None)),
+  List(genDialect(Some(false)), genDialect(Some(false)), 
genDialect(None)),
+  List(genDialect(Some(true)), genDialect(Some(true))),
+  List(genDialect(Some(false)), genDialect(Some(false))),
+  List(genDialect(None), genDialect(None))
+)
+
+val expectedCascading = Seq(Some(true), Some(true), None, Some(true), 
Some(false), None)
+
+dialectCombination.zip(expectedCascading).foreach { case (dialects, 
cascading) =>
--- End diff --

Could we combine `dialectCombination` and `expectedCascading` together? Or 
we can create a separate helper function?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadi...

2017-09-23 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/19286#discussion_r140638141
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/jdbc/AggregatedDialect.scala ---
@@ -43,6 +43,17 @@ private class AggregatedDialect(dialects: 
List[JdbcDialect]) extends JdbcDialect
   }
 
   override def isCascadingTruncateTable(): Option[Boolean] = {
-dialects.flatMap(_.isCascadingTruncateTable()).reduceOption(_ || _)
+// If any dialect claims cascading truncate, this dialect is also 
cascading truncate.
+// Otherwise, if any dialect has unknown cascading truncate, this 
dialect is also unknown.
+val cascading = 
dialects.flatMap(_.isCascadingTruncateTable()).reduceOption(_ || _)
+if (cascading.getOrElse(false)) {
+  cascading
+} else {
+  if (dialects.exists(_.isCascadingTruncateTable().isEmpty)) {
--- End diff --

combine line 51 and 52?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19333: [SPARK-22109][SQL][BRANCH-2.2] Resolves type conflicts b...

2017-09-23 Thread ueshin

Github user ueshin commented on the issue:

https://github.com/apache/spark/pull/19333
  
@HyukjinKwon Could you close this PR?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17743: [SPARK-20448][DOCS] Document how FileInputDStream works ...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17743
  
**[Test build #3934 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3934/testReport)**
 for PR 17743 at commit 
[`1e620ce`](https://github.com/apache/spark/commit/1e620ceb7b5eb0df6df83525366ebc1074f8e8ce).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19266: [SPARK-22033][CORE] BufferHolder, other size chec...

2017-09-23 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19266


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...

2017-09-23 Thread srowen

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19266
  
Merged to master


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19320: [SPARK-22099] The 'job ids' list style needs to b...

2017-09-23 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19320


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19331: [SPARK-22109][SQL] Resolves type conflicts between strin...

2017-09-23 Thread ueshin

Github user ueshin commented on the issue:

https://github.com/apache/spark/pull/19331
  
@HyukjinKwon Could you send a pr to backport this to branch-2.2 please?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18924: [SPARK-14371] [MLLIB] OnlineLDAOptimizer should not coll...

2017-09-23 Thread akopich

Github user akopich commented on the issue:

https://github.com/apache/spark/pull/18924
  
@WeichenXu123, thanks for creating Jira. Yes, sure I will work on it. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19330: Orderable MapType

2017-09-23 Thread tejasapatil

Github user tejasapatil commented on the issue:

https://github.com/apache/spark/pull/19330
  
@hvanhovell : based on [your comment over the 
jira](https://issues.apache.org/jira/browse/SPARK-18134?focusedCommentId=15693519=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15693519),
 it seemed that the approach to be used is yet to be finalised. Are we moving 
ahead with this approach ?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19333: [SPARK-22109][SQL][BRANCH-2.2] Resolves type conflicts b...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19333
  
**[Test build #82119 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82119/testReport)**
 for PR 19333 at commit 
[`42fa83c`](https://github.com/apache/spark/commit/42fa83caab5914a089d282e0c5009f7d0db42c46).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19333: [SPARK-22109][SQL][BRANCH-2.2] Resolves type conflicts b...

2017-09-23 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19333
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82119/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19277: [SPARK-22058][CORE]the BufferedInputStream will not be c...

2017-09-23 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19277
  
**[Test build #3933 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3933/testReport)**
 for PR 19277 at commit 
[`2e5f21a`](https://github.com/apache/spark/commit/2e5f21a1a89bbf69431f918d3043ff6b58420dd8).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19333: [SPARK-22109][SQL][BRANCH-2.2] Resolves type conflicts b...

2017-09-23 Thread ueshin

Github user ueshin commented on the issue:

https://github.com/apache/spark/pull/19333
  
Thanks! merging to branch-2.2.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19323: [SPARK-22092] Reallocation in OffHeapColumnVector.reserv...

2017-09-23 Thread hvanhovell

Github user hvanhovell commented on the issue:

https://github.com/apache/spark/pull/19323
  
@ala can you close this one? The merge script does not do that for 
backports.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19320: [SPARK-22099] The 'job ids' list style needs to be chang...

2017-09-23 Thread srowen

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19320
  
Merged to master


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19331: [SPARK-22109][SQL] Resolves type conflicts between strin...

2017-09-23 Thread ueshin

Github user ueshin commented on the issue:

https://github.com/apache/spark/pull/19331
  
LGTM.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

1 2 >

1 - 100 of 171 matches

Mail list logo