date:20160228

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190077736
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190077740
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52167/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190077607
  
**[Test build #52167 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52167/consoleFull)**
 for PR 11389 at commit 
[`9ca920b`](https://github.com/apache/spark/commit/9ca920bbc37dd53697b214eabfa413d61f09d682).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13309][SQL] Fix type inference issue wi...

2016-02-28 Thread tanwanirahul

Github user tanwanirahul commented on the pull request:

https://github.com/apache/spark/pull/11194#issuecomment-190074779
  
@rxin @HyukjinKwon thank you.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12941][SQL][BRANCH-1.4] Spark-SQL JDBC ...

2016-02-28 Thread thomastechs

Github user thomastechs commented on a diff in the pull request:

https://github.com/apache/spark/pull/10912#discussion_r54375435
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala 
---
@@ -445,4 +445,10 @@ class JDBCSuite extends SparkFunSuite with 
BeforeAndAfter {
 assert(agg.getCatalystType(1, "", 1, null) == Some(StringType))
   }
 
+  test("OracleDialect type mapping") {
+val oracleDialect = JdbcDialects.get("jdbc:oracle://127.0.0.1/db")
+assert(oracleDialect.getJDBCType(StringType).
+  map(_.databaseTypeDefinition).get == "VARCHAR2(255)")
+  }
+
--- End diff --

This is similar to testcase written for MySQL,for branch 1.4; which is not 
actually spinning up any MySQL or Oracle instance; but for the test suite 
purpose, as I believe.  So far, builds for this PR for branch 1.4, failed due 
to style checks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11105#issuecomment-190073652
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190073692
  
**[Test build #52176 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52176/consoleFull)**
 for PR 11389 at commit 
[`cec8442`](https://github.com/apache/spark/commit/cec8442b516cf73069a5ac8bc4c02f56618b9cc5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11105#issuecomment-190073653
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52166/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11105#issuecomment-190073389
  
**[Test build #52166 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52166/consoleFull)**
 for PR 11105 at commit 
[`fc3e39e`](https://github.com/apache/spark/commit/fc3e39e7105648f0a28d8707978783fba3534b7b).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread HyukjinKwon

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190073355
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-529] [core] [yarn] Add type-safe config...

2016-02-28 Thread rxin

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/10205#issuecomment-190073009
  
@vanzin

since this pr has changed quite a bit, can you submit the refactored config 
interface as a separate pr and let's work on the details of the interface 
there? I like the builder pattern more too.





---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-529] [core] [yarn] Add type-safe config...

2016-02-28 Thread rxin

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/10205#discussion_r54375242
  
--- Diff: core/src/main/scala/org/apache/spark/config/ConfigBuilder.scala 
---
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.config
--- End diff --

can we move this into util.config? really want to cut down the number of 
top level packages.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread HyukjinKwon

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190072703
  
test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13309][SQL] Fix type inference issue wi...

2016-02-28 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11194


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190072616
  
**[Test build #52175 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52175/consoleFull)**
 for PR 11389 at commit 
[`fea9df8`](https://github.com/apache/spark/commit/fea9df82bf37514965e5978095ec0e9c129f1ebe).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13167][SQL] Include rows with null valu...

2016-02-28 Thread rxin

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/11063#discussion_r54375174
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
 ---
@@ -75,12 +83,63 @@ private[sql] object JDBCRelation {
 }
 ans.toArray
   }
+
+  /**
+   * Returns null value predicate if the column is nullable, otherwise 
empty string. Uses
+   * nullable information available in the catalyst schema generated for 
the source table.
+   * This method avoids querying the database metadata to find if the 
unquoted column names
+   * are interpreted as uppercase or lower case. In rare cases when the 
schema has on column
+   * with uppercase name, and another one with lowercase, it will return 
null value
+   * partition predicate if either one of the column is nullable.
+   *
+   * @param column name of the column
+   * @param url the jdbc url of the database
+   * @param schema table's Catalyst schema
+   * @return  null value predicate or empty string.
+   */
+  private def getParitionNullValuePredicate(column: String,
+url: String, schema: StructType): String = {
--- End diff --

Sorry for the delay. I'd definitely just go for the simpler approach (i.e. 
always put the null in 1st partition), and only make it more complicated in the 
future if it is big enough of a problem.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13309][SQL] Fix type inference issue wi...

2016-02-28 Thread rxin

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/11194#issuecomment-190072430
  
Sorry for the delay. I'm merging this in master. Thanks!



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

2016-02-28 Thread holdenk

Github user holdenk commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190072364
  
The HiveComptability test seems likely unrelated (happened in another 
unrelated PR at the same time). Will double check in the morning.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13535] [SQL] Fix Analysis Exceptions wh...

2016-02-28 Thread gatorsmile

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/11415#issuecomment-190072232
  
@cloud-fan Could you take a look at this? Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11316] coalesce doesn't handle UnionRDD...

2016-02-28 Thread rxin

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/11327#discussion_r54375032
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -192,7 +192,8 @@ private class PartitionCoalescer(maxPartitions: Int, 
prev: RDD[_], balanceSlack:
 def resetIterator(): Iterator[(String, Partition)] = {
   val iterators = (0 to 2).map( x =>
 prev.partitions.iterator.flatMap(p => {
-  if (currPrefLocs(p).size > x) Some((currPrefLocs(p)(x), p)) else 
None
+  val locs = currPrefLocs(p)
+  if (locs.size > x) Some((locs(x), p)) else None
--- End diff --

It's not size vs length. It's seq.size can sometimes be O(n).

For size vs length, we should use length if it is a string or an array, but 
size if it is a collection.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190072167
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190072170
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52165/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7286] [SQL] Deprecate !== in favour of ...

2016-02-28 Thread rxin

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/9925#issuecomment-190072160
  
I think we should still consider it -- just haven't had time to think about 
it more yet.

Can we get somebody from the scala team to also chime in here? 

cc @dragos


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13535] [SQL] Fix Analysis Exceptions wh...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11415#issuecomment-190072039
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52169/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190072019
  
**[Test build #52165 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52165/consoleFull)**
 for PR 11423 at commit 
[`d6b3fb0`](https://github.com/apache/spark/commit/d6b3fb0c229d576fb1382c4356bd60b3f9ecfcac).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13548][BUILD] Move tags and unsafe modu...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11426#issuecomment-190071993
  
**[Test build #52174 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52174/consoleFull)**
 for PR 11426 at commit 
[`0c967cc`](https://github.com/apache/spark/commit/0c967cc95ce8b709a7c06ec442f9991fe40d9b4e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13535] [SQL] Fix Analysis Exceptions wh...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11415#issuecomment-190072038
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13548][BUILD] Move tags and unsafe modu...

2016-02-28 Thread rxin

GitHub user rxin opened a pull request:

https://github.com/apache/spark/pull/11426

[SPARK-13548][BUILD] Move tags and unsafe modules into common

## What changes were proposed in this pull request?
This patch moves tags and unsafe modules into common directory to remove 2 
top level non-user-facing directories.

## How was this patch tested?
Jenkins should suffice.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rxin/spark SPARK-13548

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11426.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11426


commit 0c967cc95ce8b709a7c06ec442f9991fe40d9b4e
Author: Reynold Xin 
Date:   2016-02-29T07:10:31Z

[SPARK-13548][BUILD] Move tags and unsafe modules into common




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13535] [SQL] Fix Analysis Exceptions wh...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11415#issuecomment-190071800
  
**[Test build #52169 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52169/consoleFull)**
 for PR 11415 at commit 
[`c1f31d1`](https://github.com/apache/spark/commit/c1f31d1f369f924866dda6fcadfd0099ebee8c5a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13548][BUILD] Move tags and unsafe modu...

2016-02-28 Thread rxin

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/11426#issuecomment-190071933
  
cc @srowen are there other things we can easily do?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12941][SQL][BRANCH-1.4] Spark-SQL JDBC ...

2016-02-28 Thread yhuai

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/10912#discussion_r54374533
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala 
---
@@ -445,4 +445,10 @@ class JDBCSuite extends SparkFunSuite with 
BeforeAndAfter {
 assert(agg.getCatalystType(1, "", 1, null) == Some(StringType))
   }
 
+  test("OracleDialect type mapping") {
+val oracleDialect = JdbcDialects.get("jdbc:oracle://127.0.0.1/db")
+assert(oracleDialect.getJDBCType(StringType).
+  map(_.databaseTypeDefinition).get == "VARCHAR2(255)")
+  }
+
--- End diff --

This test does not work, right? If so, can we remove it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Merge pull request #1 from apache/master

2016-02-28 Thread anandab

Github user anandab closed the pull request at:

https://github.com/apache/spark/pull/11425


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Merge pull request #1 from apache/master

2016-02-28 Thread anandab

GitHub user anandab opened a pull request:

https://github.com/apache/spark/pull/11425

Merge pull request #1 from apache/master

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)


## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)


(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)


pull 1

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anandab/spark master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11425.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11425


commit 1b394d930428b2161cb6a50d008ab19020bf9961
Author: Ananda Bhattacharya 
Date:   2016-02-29T05:57:22Z

Merge pull request #1 from apache/master

pull 1




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13540][SQL] Supports using nested class...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11421#issuecomment-190065989
  
**[Test build #52173 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52173/consoleFull)**
 for PR 11421 at commit 
[`4ce71ce`](https://github.com/apache/spark/commit/4ce71ce9c8fbe64d158570886c25567801dbe104).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13540][SQL] Supports using nested class...

2016-02-28 Thread liancheng

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/11421#issuecomment-190064821
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13544] [SQL] Rewrite/Propagate Constrai...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11422#issuecomment-190064421
  
**[Test build #52172 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52172/consoleFull)**
 for PR 11422 at commit 
[`ecf4b08`](https://github.com/apache/spark/commit/ecf4b087a4a059b1c66cd2deebfd103786b3f41e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13545] [MLlib] [PySpark] Make MLlib Log...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11424#issuecomment-190063692
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52170/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13545] [MLlib] [PySpark] Make MLlib Log...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11424#issuecomment-190063691
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13545] [MLlib] [PySpark] Make MLlib Log...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11424#issuecomment-190063635
  
**[Test build #52170 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52170/consoleFull)**
 for PR 11424 at commit 
[`fc370c0`](https://github.com/apache/spark/commit/fc370c0c3fba12af42551d4d71043cb54e3fde71).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190060074
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52163/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190060071
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190059745
  
**[Test build #52163 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52163/consoleFull)**
 for PR 11389 at commit 
[`9ca920b`](https://github.com/apache/spark/commit/9ca920bbc37dd53697b214eabfa413d61f09d682).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13540][SQL] Supports using nested class...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11421#issuecomment-190059550
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52164/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13540][SQL] Supports using nested class...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11421#issuecomment-190059549
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13540][SQL] Supports using nested class...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11421#issuecomment-190059289
  
**[Test build #52164 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52164/consoleFull)**
 for PR 11421 at commit 
[`28e1139`](https://github.com/apache/spark/commit/28e11390d52fc9d4151bb3eee1f86951f093b972).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13540][SQL] Supports using nested class...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11421#issuecomment-190058759
  
**[Test build #52171 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52171/consoleFull)**
 for PR 11421 at commit 
[`4ce71ce`](https://github.com/apache/spark/commit/4ce71ce9c8fbe64d158570886c25567801dbe104).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13544] [SQL] Rewrite/Propagate Constrai...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11422#issuecomment-190057233
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52162/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13544] [SQL] Rewrite/Propagate Constrai...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11422#issuecomment-190057230
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13544] [SQL] Rewrite/Propagate Constrai...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11422#issuecomment-190057138
  
**[Test build #52162 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52162/consoleFull)**
 for PR 11422 at commit 
[`9c17ebe`](https://github.com/apache/spark/commit/9c17ebefe58557850f3a6c6bb6232ca7f2cf7753).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13540][SQL] Supports using nested class...

2016-02-28 Thread cloud-fan

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/11421#discussion_r54371448
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala 
---
@@ -621,12 +621,22 @@ class DatasetSuite extends QueryTest with 
SharedSQLContext {
   ds.filter(_ => true),
   Some(1), Some(2), Some(3))
   }
+
+  test("SPARK-13540 add Scala object to OuterScopes") {
--- End diff --

the test name should be updated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13540][SQL] Supports using nested class...

2016-02-28 Thread cloud-fan

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/11421#issuecomment-190056759
  
LGTM except some minor comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12941][SQL][BRANCH-1.4] Spark-SQL JDBC ...

2016-02-28 Thread thomastechs

Github user thomastechs commented on the pull request:

https://github.com/apache/spark/pull/10912#issuecomment-190056592
  
@yhuaiThe style check error for this PR has been fixed. This is for 
branch 1.4


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13540][SQL] Supports using nested class...

2016-02-28 Thread cloud-fan

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/11421#discussion_r54371417
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ---
@@ -559,7 +561,13 @@ class Analyzer(
   }
 
   resolveExpression(unbound, LocalRelation(attributes), throws = true) 
transform {
-case n: NewInstance if n.outerPointer.isEmpty && 
n.cls.isMemberClass =>
+case n: NewInstance
+  // If this is an inner class of another class, register the 
outer object in `OuterScopes`.
+  // Note that static inner classes (e.g., inner classes within 
Scala objects. don't need
--- End diff --

nit: `(e.g. inner classes within Scala objects)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190056031
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52161/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190056027
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190055689
  
**[Test build #52161 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52161/consoleFull)**
 for PR 11389 at commit 
[`9ca920b`](https://github.com/apache/spark/commit/9ca920bbc37dd53697b214eabfa413d61f09d682).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13545] [MLlib] [PySpark] Make MLlib LR'...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11424#issuecomment-190055457
  
**[Test build #52170 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52170/consoleFull)**
 for PR 11424 at commit 
[`fc370c0`](https://github.com/apache/spark/commit/fc370c0c3fba12af42551d4d71043cb54e3fde71).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13545] [MLlib] [PySpark] Make MLlib LR'...

2016-02-28 Thread yanboliang

GitHub user yanboliang opened a pull request:

https://github.com/apache/spark/pull/11424

[SPARK-13545] [MLlib] [PySpark] Make MLlib LR's default parameters 
consistent in Scala and Python

## What changes were proposed in this pull request?
Make MLlib LR's default parameters consistent in Scala and Python.
## How was this patch tested?
No new tests, should pass current tests.




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yanboliang/spark spark-13545

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11424.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11424


commit fc370c0c3fba12af42551d4d71043cb54e3fde71
Author: Yanbo Liang 
Date:   2016-02-29T05:32:58Z

Make MLlib LR's default parameters consistent in Scala and Python




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13535] [SQL] Fix Analysis Exceptions wh...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11415#issuecomment-190054949
  
**[Test build #52169 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52169/consoleFull)**
 for PR 11415 at commit 
[`c1f31d1`](https://github.com/apache/spark/commit/c1f31d1f369f924866dda6fcadfd0099ebee8c5a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13529][BUILD] Move network/* modules in...

2016-02-28 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11409


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13537][SQL] Fix readBytes in Vectorized...

2016-02-28 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11418


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13233][SQL] Python Dataset (basic versi...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11347#issuecomment-190043823
  
**[Test build #52168 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52168/consoleFull)**
 for PR 11347 at commit 
[`9beffc6`](https://github.com/apache/spark/commit/9beffc69fc829b92413f54f89fc55bb59d2a2b56).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190043809
  
**[Test build #52167 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52167/consoleFull)**
 for PR 11389 at commit 
[`9ca920b`](https://github.com/apache/spark/commit/9ca920bbc37dd53697b214eabfa413d61f09d682).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13537][SQL] Fix readBytes in Vectorized...

2016-02-28 Thread rxin

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/11418#issuecomment-190042880
  
Thanks - I've merged this in master.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread HyukjinKwon

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190041264
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread HyukjinKwon

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190041162
  
As this passes sometimes (e.g. https://github.com/apache/spark/pull/11016), 
I well restart.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11105#issuecomment-190041051
  
**[Test build #52166 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52166/consoleFull)**
 for PR 11105 at commit 
[`fc3e39e`](https://github.com/apache/spark/commit/fc3e39e7105648f0a28d8707978783fba3534b7b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190040805
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190040806
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52160/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190040726
  
**[Test build #52160 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52160/consoleFull)**
 for PR 11389 at commit 
[`9ca920b`](https://github.com/apache/spark/commit/9ca920bbc37dd53697b214eabfa413d61f09d682).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-02-28 Thread holdenk

Github user holdenk commented on the pull request:

https://github.com/apache/spark/pull/11105#issuecomment-190040644
  
HiveContext test failure seems unrelated, jenkins retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190040613
  
**[Test build #52165 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52165/consoleFull)**
 for PR 11423 at commit 
[`d6b3fb0`](https://github.com/apache/spark/commit/d6b3fb0c229d576fb1382c4356bd60b3f9ecfcac).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...

2016-02-28 Thread sarutak

Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/11301#issuecomment-190040447
  
O.K. I'll inspect this change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13108][SQL] Validate ascii compatible e...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11016#issuecomment-190040446
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13108][SQL] Validate ascii compatible e...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11016#issuecomment-190040449
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52157/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

2016-02-28 Thread holdenk

GitHub user holdenk opened a pull request:

https://github.com/apache/spark/pull/11423

[SPARK-13398][STREAMING] Move away from thread pool task support java 
forkjoin

## What changes were proposed in this pull request?

Remove old deprecated ThreadPoolExecutor and replace with ExecutionContext 
using a ForkJoinPool. The downside of this is that scala's ForkJoinPool doesn't 
give us a way to specify the thread pool name (and is also deprecated in 2.12). 
We can use the Java version of ForkJoinPool instead, although we need supply 
the custom factory instead. One other implicit change that happens is the old 
ExecutionContext would have reported a different default parallelism since it 
used system parallelism rather than threadpool parallelism (this was likely not 
intended but also likely not a huge difference).

The previous version of this PR attempted to use an execution context 
constructed on the ThreadPool (but not the deprecated ThreadPoolExecutor class) 
so as to keep the ability to have human readable named threads but this 
reported system parallelism.

## How was this patch tested?

unit tests: streaming/testOnly org.apache.spark.streaming.util.*


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/holdenk/spark 
SPARK-13398-move-away-from-ThreadPoolTaskSupport-java-forkjoin

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11423.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11423


commit 68df46a1093132ad3d5eef3e7a3d669105083236
Author: Holden Karau 
Date:   2016-02-21T07:08:45Z

Change FileWriteAheadLogger to not depend on deprecated ThreadPoolContext

commit 9991ebce07b6fabe122719ba1c4690e733297b18
Author: Holden Karau 
Date:   2016-02-21T07:13:40Z

Remove now unused import

commit f2e6a25ae2581e9b6cc43950c1302c26a8602c2a
Author: Holden Karau 
Date:   2016-02-21T07:19:27Z

Update the comment to point out if the thread pool is smaller than 8, that 
is the number of objects kept in memory.

commit ed9ff855871bfef69b39b1b9b791d1ce487093bd
Author: Holden Karau 
Date:   2016-02-22T01:07:56Z

Use a ForkJoinPool as the base

commit 296130cd262a3314349ced5b310a1b99130992e9
Author: Holden Karau 
Date:   2016-02-29T00:32:10Z

Merge branch 'master' into 
SPARK-13398-move-away-from-ThreadPoolTaskSupport-java-forkjoin

commit 7218567c146126b12a213117e6cf8069e33db1ea
Author: Holden Karau 
Date:   2016-02-29T02:50:06Z

Use a Java based ForkJoinPool

commit d6b3fb0c229d576fb1382c4356bd60b3f9ecfcac
Author: Holden Karau 
Date:   2016-02-29T02:59:23Z

Style fixes.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13537][SQL] Fix readBytes in Vectorized...

2016-02-28 Thread viirya

Github user viirya commented on the pull request:

https://github.com/apache/spark/pull/11418#issuecomment-190040427
  
I saw the failure in #11415 jenkins test report. Then I rerun the test 
locally to find the problematic data and do debugging with it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11105#issuecomment-190040419
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52155/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11105#issuecomment-190040417
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11105#issuecomment-190040323
  
**[Test build #52155 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52155/consoleFull)**
 for PR 11105 at commit 
[`fc3e39e`](https://github.com/apache/spark/commit/fc3e39e7105648f0a28d8707978783fba3534b7b).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13108][SQL] Validate ascii compatible e...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11016#issuecomment-190040314
  
**[Test build #52157 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52157/consoleFull)**
 for PR 11016 at commit 
[`a6f6023`](https://github.com/apache/spark/commit/a6f60234bc82834490271e57842df0775a0acf8b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13123][SQL] Implement whole state codeg...

2016-02-28 Thread nongli

Github user nongli commented on a diff in the pull request:

https://github.com/apache/spark/pull/11359#discussion_r54369578
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/Sort.scala 
---
@@ -93,4 +97,75 @@ case class Sort(
   sortedIterator
 }
   }
+
+  override def upstreams(): Seq[RDD[InternalRow]] = {
+child.asInstanceOf[CodegenSupport].upstreams()
+  }
+
+  // Name of sorter variable used in codegen.
+  private var sorterVariable: String = _
+
+  override protected def doProduce(ctx: CodegenContext): String = {
+val needToSort = ctx.freshName("needToSort")
+ctx.addMutableState("boolean", needToSort, s"$needToSort = true;")
+
+
+// Initialize the class member variables. This includes the instance 
of the Sorter and
+// the iterator to return sorted rows.
+val thisPlan = ctx.addReferenceObj("plan", this)
+sorterVariable = ctx.freshName("sorter")
+ctx.addMutableState(classOf[UnsafeExternalRowSorter].getName, 
sorterVariable,
+  s"$sorterVariable = $thisPlan.createSorter();")
+val metrics = ctx.freshName("metrics")
+ctx.addMutableState(classOf[TaskMetrics].getName, metrics,
+  s"$metrics = org.apache.spark.TaskContext.get().taskMetrics();")
+val sortedIterator = ctx.freshName("sortedIter")
+ctx.addMutableState("scala.collection.Iterator", 
sortedIterator, "")
+
+val addToSorter = ctx.freshName("addToSorter")
+ctx.addNewFunction(addToSorter,
+  s"""
+| private void $addToSorter() throws java.io.IOException {
+|   ${child.asInstanceOf[CodegenSupport].produce(ctx, this)}
+| }
+  """.stripMargin.trim)
+
+val outputRow = ctx.freshName("outputRow")
+val dataSize = metricTerm(ctx, "dataSize")
+val spillSize = metricTerm(ctx, "spillSize")
+val spillSizeBefore = ctx.freshName("spillSizeBefore")
+ctx.addMutableState(classOf[Long].getName, spillSizeBefore, "")
+s"""
+   | if ($needToSort) {
+   |   $addToSorter();
+   |   $spillSizeBefore = $metrics.memoryBytesSpilled();
+   |   $sortedIterator = $sorterVariable.sort();
+   |   $dataSize.add($sorterVariable.getPeakMemoryUsage());
+   |   $spillSize.add($metrics.memoryBytesSpilled() - 
$spillSizeBefore);
+   |   
$metrics.incPeakExecutionMemory($sorterVariable.getPeakMemoryUsage());
+   |   $needToSort = false;
+   | }
+   |
+   | while ($sortedIterator.hasNext()) {
+   |   UnsafeRow $outputRow = (UnsafeRow)$sortedIterator.next();
+   |   ${consume(ctx, null, outputRow)}
+   |   if (shouldStop()) return;
+   | }
+ """.stripMargin.trim
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode]): 
String = {
+val colExprs = child.output.zipWithIndex.map { case (attr, i) =>
+  BoundReference(i, attr.dataType, attr.nullable)
+}
+
+ctx.currentVars = input
+val code = GenerateUnsafeProjection.createCode(ctx, colExprs, 
useSubexprElimination = false)
--- End diff --

Any reason to explicilty set subexpr to false?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13123][SQL] Implement whole state codeg...

2016-02-28 Thread nongli

Github user nongli commented on the pull request:

https://github.com/apache/spark/pull/11359#issuecomment-190040188
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13484][SQL] Prevent illegal NULL propag...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11371#issuecomment-190040111
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13484][SQL] Prevent illegal NULL propag...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11371#issuecomment-190040114
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52159/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13123][SQL] Implement whole state codeg...

2016-02-28 Thread nongli

Github user nongli commented on a diff in the pull request:

https://github.com/apache/spark/pull/11359#discussion_r54369543
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/Sort.scala 
---
@@ -93,4 +97,75 @@ case class Sort(
   sortedIterator
 }
   }
+
+  override def upstreams(): Seq[RDD[InternalRow]] = {
+child.asInstanceOf[CodegenSupport].upstreams()
+  }
+
+  // Name of sorter variable used in codegen.
+  private var sorterVariable: String = _
+
+  override protected def doProduce(ctx: CodegenContext): String = {
+val needToSort = ctx.freshName("needToSort")
+ctx.addMutableState("boolean", needToSort, s"$needToSort = true;")
+
+
+// Initialize the class member variables. This includes the instance 
of the Sorter and
+// the iterator to return sorted rows.
+val thisPlan = ctx.addReferenceObj("plan", this)
+sorterVariable = ctx.freshName("sorter")
+ctx.addMutableState(classOf[UnsafeExternalRowSorter].getName, 
sorterVariable,
+  s"$sorterVariable = $thisPlan.createSorter();")
+val metrics = ctx.freshName("metrics")
+ctx.addMutableState(classOf[TaskMetrics].getName, metrics,
+  s"$metrics = org.apache.spark.TaskContext.get().taskMetrics();")
+val sortedIterator = ctx.freshName("sortedIter")
+ctx.addMutableState("scala.collection.Iterator", 
sortedIterator, "")
+
+val addToSorter = ctx.freshName("addToSorter")
+ctx.addNewFunction(addToSorter,
+  s"""
+| private void $addToSorter() throws java.io.IOException {
+|   ${child.asInstanceOf[CodegenSupport].produce(ctx, this)}
+| }
+  """.stripMargin.trim)
+
+val outputRow = ctx.freshName("outputRow")
+val dataSize = metricTerm(ctx, "dataSize")
+val spillSize = metricTerm(ctx, "spillSize")
+val spillSizeBefore = ctx.freshName("spillSizeBefore")
--- End diff --

This can just be a local var. Just remove the ".addMutableState" below and 
fix line 141.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13484][SQL] Prevent illegal NULL propag...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11371#issuecomment-190040034
  
**[Test build #52159 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52159/consoleFull)**
 for PR 11371 at commit 
[`f1718d6`](https://github.com/apache/spark/commit/f1718d6932122b9020ba9b86254f34b953a83e99).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13537][SQL] Fix readBytes in Vectorized...

2016-02-28 Thread nongli

Github user nongli commented on the pull request:

https://github.com/apache/spark/pull/11418#issuecomment-190038996
  
LGTM

Thanks for fixing this. Just out of curiosity, how did you find this 
initially?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13540][SQL] Supports using nested class...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11421#issuecomment-190038814
  
**[Test build #52164 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52164/consoleFull)**
 for PR 11421 at commit 
[`28e1139`](https://github.com/apache/spark/commit/28e11390d52fc9d4151bb3eee1f86951f093b972).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13540][SQL] Supports using nested class...

2016-02-28 Thread liancheng

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/11421#issuecomment-190038571
  
Thanks for the review! Comments addressed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13535] [SQL] Fix Analysis Exceptions wh...

2016-02-28 Thread gatorsmile

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/11415#issuecomment-190035820
  
yeah, but this PR is so unlucky. 100% failure : ) 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13535] [SQL] Fix Analysis Exceptions wh...

2016-02-28 Thread viirya

Github user viirya commented on the pull request:

https://github.com/apache/spark/pull/11415#issuecomment-190035401
  
@gatorsmile the failed test is not related to this, I have sumbitted a pr 
to fix it. Before it is merged, the test you ask to retest will be failed very 
frequently.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread HyukjinKwon

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190034727
  
@yhuai Could I ask that you have any clue on this? I think this is related 
with whole-code generation. This is happening some builds such as 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52154/consoleFull,
 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52151/consoleFull
 and 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52153/consoleFull.

I could not find any clues until now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13544] [SQL] Rewrite/Propagate Constrai...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11422#issuecomment-190034692
  
**[Test build #52162 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52162/consoleFull)**
 for PR 11422 at commit 
[`9c17ebe`](https://github.com/apache/spark/commit/9c17ebefe58557850f3a6c6bb6232ca7f2cf7753).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190034693
  
**[Test build #52163 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52163/consoleFull)**
 for PR 11389 at commit 
[`9ca920b`](https://github.com/apache/spark/commit/9ca920bbc37dd53697b214eabfa413d61f09d682).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13535] [SQL] Fix Analysis Exceptions wh...

2016-02-28 Thread gatorsmile

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/11415#issuecomment-190034027
  
retest this please



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread HyukjinKwon

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190033592
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11517][SQL]Calc partitions in parallel ...

2016-02-28 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9483#issuecomment-190033570
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13509][SPARK-13507][SQL] Support for wr...

2016-02-28 Thread HyukjinKwon

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/11389#issuecomment-190033580
  
Hm... It looks a bit weird. `ParquetHadoopFsRelationSuite`  the tests about 
types ("test all data types - ...") keep failing. This is also happening at 
other PRs I made.

I thought I wanted to submit a hot-fix but I found it actually works okay 
in my local.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13544] [SQL] Rewrite/Propagate Constrai...

2016-02-28 Thread gatorsmile

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/11422

[SPARK-13544] [SQL] Rewrite/Propagate Constraints for Aliases in Aggregate 
and Windows

 What changes were proposed in this pull request?

After analysis by Analyzer, three operators could have alias. They are 
`Project`, `Aggregate` and `Window`. So far, we only rewrite and propagate 
constraints if `Alias` is defined in `Project`. This PR is to resolve this 
issue in `Aggregate` and `Window`.

 How was this patch tested?

Added a test case for `Aggregate` in `ConstraintPropagationSuite`. We do 
not have a DSL for `Window`. It is a little bit tedious to add such a similar 
case for `Windows`. Could we skip it? : )

@marmbrus @sameeragarwal 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark validConstraintsInUnaryNodes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11422.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11422


commit 9c17ebefe58557850f3a6c6bb6232ca7f2cf7753
Author: gatorsmile 
Date:   2016-02-29T04:12:52Z

Propagate constraints for Aliases in Aggregate and Windows




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

1 2 3 >

1 - 100 of 219 matches

Mail list logo