date:20171021

[GitHub] spark issue #19548: [SPARK-22303][SQL] Handle Oracle specific jdbc types in ...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19548
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82951/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19548: [SPARK-22303][SQL] Handle Oracle specific jdbc types in ...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19548
  
**[Test build #82951 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82951/testReport)**
 for PR 19548 at commit 
[`51c616c`](https://github.com/apache/spark/commit/51c616c6e501ad539f4e18cf604250a66edf1a2e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17100: [SPARK-13947][SQL] PySpark DataFrames: The error ...

2017-10-21 Thread viirya

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/17100#discussion_r146121917
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
 ---
@@ -270,12 +273,25 @@ trait CheckAnalysis extends PredicateHelper {
 
 operator match {
   case o if o.children.nonEmpty && o.missingInput.nonEmpty =>
+val resolver = conf.resolver
+val commonAttrs = o.missingInput.filter(x =>
+  o.inputSet.exists(y => resolver(x.name, y.name)))
 val missingAttributes = o.missingInput.mkString(",")
-val input = o.inputSet.mkString(",")
+val availableAttributes = o.inputSet.mkString(",")
+val repeatedNameHint = if (commonAttrs.size > 0) {
+  val commonNames = commonAttrs.map(_.name).mkString(",")
+  s"""\n|Observe that attribute(s) $commonNames appear in two
--- End diff --

The new line `\n` is intentional?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17100: [SPARK-13947][SQL] PySpark DataFrames: The error ...

2017-10-21 Thread viirya

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/17100#discussion_r146121907
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
 ---
@@ -270,12 +273,25 @@ trait CheckAnalysis extends PredicateHelper {
 
 operator match {
   case o if o.children.nonEmpty && o.missingInput.nonEmpty =>
+val resolver = conf.resolver
+val commonAttrs = o.missingInput.filter(x =>
+  o.inputSet.exists(y => resolver(x.name, y.name)))
 val missingAttributes = o.missingInput.mkString(",")
-val input = o.inputSet.mkString(",")
+val availableAttributes = o.inputSet.mkString(",")
+val repeatedNameHint = if (commonAttrs.size > 0) {
+  val commonNames = commonAttrs.map(_.name).mkString(",")
+  s"""\n|Observe that attribute(s) $commonNames appear in two
+  |different datasets, with the same name"""
--- End diff --

Missing a `.` at the end of sentence. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17100: [SPARK-13947][SQL] PySpark DataFrames: The error ...

2017-10-21 Thread viirya

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/17100#discussion_r146121901
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
 ---
@@ -270,12 +273,25 @@ trait CheckAnalysis extends PredicateHelper {
 
 operator match {
   case o if o.children.nonEmpty && o.missingInput.nonEmpty =>
+val resolver = conf.resolver
+val commonAttrs = o.missingInput.filter(x =>
+  o.inputSet.exists(y => resolver(x.name, y.name)))
 val missingAttributes = o.missingInput.mkString(",")
-val input = o.inputSet.mkString(",")
+val availableAttributes = o.inputSet.mkString(",")
+val repeatedNameHint = if (commonAttrs.size > 0) {
+  val commonNames = commonAttrs.map(_.name).mkString(",")
+  s"""\n|Observe that attribute(s) $commonNames appear in two
--- End diff --

I'm not sure if this is always the case. Some failures in the query plan 
analysis can cause missing attributes, I mean.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19549: [SPARK-22327][SPARKR][TEST] check for version warning

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19549
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19549: [SPARK-22327][SPARKR][TEST] check for version warning

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19549
  
**[Test build #82952 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82952/testReport)**
 for PR 19549 at commit 
[`53d0603`](https://github.com/apache/spark/commit/53d0603ed8c4083f984f43e426533eabc6a0839b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19549: [SPARK-22327][SPARKR][TEST] check for version warning

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19549
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82952/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19550: [SPARK-22327][SPARKR][TEST][BACKPORT-2.0] check for vers...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19550
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19550: [SPARK-22327][SPARKR][TEST][BACKPORT-2.0] check for vers...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19550
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82953/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19550: [SPARK-22327][SPARKR][TEST][BACKPORT-2.0] check for vers...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19550
  
**[Test build #82953 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82953/consoleFull)**
 for PR 19550 at commit 
[`3a5b6fa`](https://github.com/apache/spark/commit/3a5b6fa891bd03f5007cc431790d3a7e769a78fc).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17100: [SPARK-13947][SQL] PySpark DataFrames: The error ...

2017-10-21 Thread viirya

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/17100#discussion_r146121699
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala
 ---
@@ -408,16 +408,28 @@ class AnalysisErrorSuite extends AnalysisTest {
 // CheckAnalysis should throw AnalysisException when Aggregate 
contains missing attribute(s)
 // Since we manually construct the logical plan at here and Sum only 
accept
 // LongType, DoubleType, and DecimalType. We use LongType as the type 
of a.
-val plan =
-  Aggregate(
+val attrA = AttributeReference("a", LongType)(exprId = ExprId(1))
+val aId = Array[String](attrA.name, attrA.exprId.id.toString)
+val otherA = AttributeReference("a", LongType)(exprId = ExprId(2))
+val otherAId = Array[String](otherA.name, otherA.exprId.id.toString)
+val bAlias = Alias(sum(attrA), "b")() :: Nil
+val plan = Aggregate(
 Nil,
-Alias(sum(AttributeReference("a", LongType)(exprId = ExprId(1))), 
"b")() :: Nil,
-LocalRelation(
-  AttributeReference("a", LongType)(exprId = ExprId(2
+bAlias,
+LocalRelation(otherA))
 
 assert(plan.resolved)
 
-assertAnalysisError(plan, "resolved attribute(s) a#1L missing from 
a#2L" :: Nil)
+val errorMsg = s"""Some resolved attribute(s) are not present among 
the available attributes
+ |for a query.
+ |${aId.mkString("#")}L is not in 
${otherAId.mkString("#")}L.
--- End diff --

Isn't `${aId.mkString("#")}L` just `attrA.toString`?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19550: [SPARK-22327][SPARKR][TEST][BACKPORT-2.0] check for vers...

2017-10-21 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/19550
  
this was the last build 
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-branch-2.0-test-sbt-hadoop-2.7/977/console


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19550: [SPARK-22327][SPARKR][TEST][BACKPORT-2.0] check for vers...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19550
  
**[Test build #82953 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82953/consoleFull)**
 for PR 19550 at commit 
[`3a5b6fa`](https://github.com/apache/spark/commit/3a5b6fa891bd03f5007cc431790d3a7e769a78fc).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19550: [SPARK-22327][SPARKR][TEST] check for version war...

2017-10-21 Thread felixcheung

GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/19550

[SPARK-22327][SPARKR][TEST] check for version warning

## What changes were proposed in this pull request?

backporting to 2.0 (firstm since it's the first branch "older than" 2.1.2)

## How was this patch tested?

manually
Jenkins, AppVeyor

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark rcranversioncheck20

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19550.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19550


commit 3a5b6fa891bd03f5007cc431790d3a7e769a78fc
Author: Felix Cheung 
Date:   2017-10-22T04:03:16Z

check for version warning




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19549: [SPARK-22327][SPARKR][TEST] check for version warning

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19549
  
**[Test build #82952 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82952/testReport)**
 for PR 19549 at commit 
[`53d0603`](https://github.com/apache/spark/commit/53d0603ed8c4083f984f43e426533eabc6a0839b).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19549: [SPARK-22327][SPARKR][TEST] check for version war...

2017-10-21 Thread felixcheung

GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/19549

[SPARK-22327][SPARKR][TEST] check for version warning

## What changes were proposed in this pull request?

Will need to port to this branch-1.6, -2.0, -2.1, -2.2

## How was this patch tested?

Jenkins, AppVeyor

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark rcranversioncheck

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19549.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19549


commit 53d0603ed8c4083f984f43e426533eabc6a0839b
Author: Felix Cheung 
Date:   2017-10-22T04:03:16Z

check for version warning




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19540: [SPARK-22319][Core] call loginUserFromKeytab before acce...

2017-10-21 Thread jiangxb1987

Github user jiangxb1987 commented on the issue:

https://github.com/apache/spark/pull/19540
  
Could you check why the test cases failed? Thanks!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19548: [SPARK-22303][SQL] Handle Oracle specific jdbc ty...

2017-10-21 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/19548#discussion_r146120958
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/jdbc/OracleDialect.scala ---
@@ -28,25 +28,28 @@ private case object OracleDialect extends JdbcDialect {
 
   override def getCatalystType(
   sqlType: Int, typeName: String, size: Int, md: MetadataBuilder): 
Option[DataType] = {
-if (sqlType == Types.NUMERIC) {
-  val scale = if (null != md) md.build().getLong("scale") else 0L
-  size match {
-// Handle NUMBER fields that have no precision/scale in special way
-// because JDBC ResultSetMetaData converts this to 0 precision and 
-127 scale
-// For more details, please see
-// https://github.com/apache/spark/pull/8780#issuecomment-145598968
-// and
-// https://github.com/apache/spark/pull/8780#issuecomment-144541760
-case 0 => Option(DecimalType(DecimalType.MAX_PRECISION, 10))
-// Handle FLOAT fields in a special way because JDBC 
ResultSetMetaData converts
-// this to NUMERIC with -127 scale
-// Not sure if there is a more robust way to identify the field as 
a float (or other
-// numeric types that do not specify a scale.
-case _ if scale == -127L => 
Option(DecimalType(DecimalType.MAX_PRECISION, 10))
-case _ => None
-  }
-} else {
-  None
+sqlType match {
+  case Types.NUMERIC =>
+val scale = if (null != md) md.build().getLong("scale") else 0L
+size match {
+  // Handle NUMBER fields that have no precision/scale in special 
way
+  // because JDBC ResultSetMetaData converts this to 0 precision 
and -127 scale
+  // For more details, please see
+  // 
https://github.com/apache/spark/pull/8780#issuecomment-145598968
+  // and
+  // 
https://github.com/apache/spark/pull/8780#issuecomment-144541760
+  case 0 => Option(DecimalType(DecimalType.MAX_PRECISION, 10))
+  // Handle FLOAT fields in a special way because JDBC 
ResultSetMetaData converts
+  // this to NUMERIC with -127 scale
+  // Not sure if there is a more robust way to identify the field 
as a float (or other
+  // numeric types that do not specify a scale.
+  case _ if scale == -127L => 
Option(DecimalType(DecimalType.MAX_PRECISION, 10))
+  case _ => None
+}
+  case -101 => Some(TimestampType) // Value for Timestamp with Time 
Zone in Oracle
+  case 100 => Some(FloatType) // Value for OracleTypes.BINARY_FLOAT
+  case 101 => Some(DoubleType) // Value for OracleTypes.BINARY_DOUBLE
--- End diff --

Question: Is the value range of `OracleTypes.BINARY_DOUBLE` identical to 
our Spark Double type?

Also, `OracleTypes.BINARY_FLOAT ` with Spark Float type


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19548: [SPARK-22303][SQL] Handle Oracle specific jdbc types in ...

2017-10-21 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/19548
  
Could you add a test case to `OracleIntegrationSuite.scala`?

Below is the instruction how to run the docker-based testsuite 
`OracleIntegrationSuite.scala `.

> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0  -Phive-thriftserver 
-Phive -DskipTests  install
> Before running dockers test . You need to set the docker env variabled. 
Following does the magic:
> eval $(docker-machine env default)
> build/mvn -Pdocker-integration-tests -pl 
:spark-docker-integration-tests_2.11  compile test


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19548: [SPARK-22303][SQL] Handle Oracle specific jdbc types in ...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19548
  
**[Test build #82951 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82951/testReport)**
 for PR 19548 at commit 
[`51c616c`](https://github.com/apache/spark/commit/51c616c6e501ad539f4e18cf604250a66edf1a2e).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19548: [SPARK-22303][SQL] Handle Oracle specific jdbc types in ...

2017-10-21 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/19548
  
ok to test


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17100: [SPARK-13947][SQL] PySpark DataFrames: The error message...

2017-10-21 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17100
  
cc @wzhfy Could you review this PR?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17100: [SPARK-13947][SQL] PySpark DataFrames: The error ...

2017-10-21 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17100#discussion_r146120797
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
 ---
@@ -270,12 +273,25 @@ trait CheckAnalysis extends PredicateHelper {
 
 operator match {
   case o if o.children.nonEmpty && o.missingInput.nonEmpty =>
+val resolver = conf.resolver
--- End diff --

We do not need to pass conf now. You can directly call `plan.conf.resolver`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19548: [SPARK-22303][SQL] Handle Oracle specific jdbc types in ...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19548
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17100: [SPARK-13947][SQL] PySpark DataFrames: The error ...

2017-10-21 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17100#discussion_r146120800
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
 ---
@@ -125,7 +128,7 @@ trait CheckAnalysis extends PredicateHelper {
 }
 
   case s: SubqueryExpression =>
-checkSubqueryExpression(operator, s)
+checkAnalysisWithConf(s.plan, conf)
--- End diff --

This is wrong. Could you revert it back?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19548: [SPARK-22303][SQL] Handle Oracle specific jdbc ty...

2017-10-21 Thread taroplus

GitHub user taroplus opened a pull request:

https://github.com/apache/spark/pull/19548

[SPARK-22303][SQL] Handle Oracle specific jdbc types in OracleDialect

TIMESTAMP (-101), BINARY_DOUBLE (101) and BINARY_FLOAT (100) are handled in 
OracleDialect

## What changes were proposed in this pull request?

When a oracle table contains columns whose type is BINARY_FLOAT or 
BINARY_DOUBLE, spark sql fails to load a table with SQLException

```
java.sql.SQLException: Unsupported type 101
 at 
org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.org$apache$spark$sql$execution$datasources$jdbc$JdbcUtils$$getCatalystType(JdbcUtils.scala:235)
 at 
org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$8.apply(JdbcUtils.scala:292)
 at 
org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$8.apply(JdbcUtils.scala:292)
 at scala.Option.getOrElse(Option.scala:121)
 at 
org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(JdbcUtils.scala:291)
 at 
org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:64)
 at 
org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.(JDBCRelation.scala:113)
 at 
org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:47)
 at 
org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:306)
 at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
 at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:146)
```

## How was this patch tested?

I updated a UT which covers type conversion test for types (-101, 100, 
101), on top of that I tested this change against actual table with those 
columns and it was able to read and write to the table. 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/taroplus/spark oracle_sql_types_101

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19548.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19548


commit 51c616c6e501ad539f4e18cf604250a66edf1a2e
Author: Kohki Nishio 
Date:   2017-10-22T02:55:28Z

Handle Oracle specific jdbc types in OracleDialect

TIMESTAMP (-101), BINARY_DOUBLE (101) and BINARY_FLOAT (100) are handled in 
OracleDialect




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19545: [SPARK-21929][SQL] Support `ALTER TABLE table_name ADD C...

2017-10-21 Thread dongjoon-hyun

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/19545
  
Thank you a lot, @gatorsmile !


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19545: [SPARK-21929][SQL] Support `ALTER TABLE table_nam...

2017-10-21 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19545


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19545: [SPARK-21929][SQL] Support `ALTER TABLE table_name ADD C...

2017-10-21 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/19545
  
Thanks! Merged to master.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17100: [SPARK-13947][SQL] PySpark DataFrames: The error message...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17100
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17100: [SPARK-13947][SQL] PySpark DataFrames: The error message...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17100
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82950/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17100: [SPARK-13947][SQL] PySpark DataFrames: The error message...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17100
  
**[Test build #82950 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82950/testReport)**
 for PR 17100 at commit 
[`6bbe496`](https://github.com/apache/spark/commit/6bbe496f9d8bf48953ba15518d5f80f0b4140ed5).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17100: [SPARK-13947][SQL] PySpark DataFrames: The error message...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17100
  
**[Test build #82950 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82950/testReport)**
 for PR 17100 at commit 
[`6bbe496`](https://github.com/apache/spark/commit/6bbe496f9d8bf48953ba15518d5f80f0b4140ed5).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19545: [SPARK-21929][SQL] Support `ALTER TABLE table_name ADD C...

2017-10-21 Thread dongjoon-hyun

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/19545
  
Itâs ready for review again, @gatorsmile. Thanks. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19514: [SPARK-21551][Python] Increase timeout for Python...

2017-10-21 Thread FRosner

Github user FRosner closed the pull request at:

https://github.com/apache/spark/pull/19514


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19545: [SPARK-21929][SQL] Support `ALTER TABLE table_name ADD C...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19545
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19545: [SPARK-21929][SQL] Support `ALTER TABLE table_name ADD C...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19545
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82949/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19545: [SPARK-21929][SQL] Support `ALTER TABLE table_name ADD C...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19545
  
**[Test build #82949 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82949/testReport)**
 for PR 19545 at commit 
[`dfc59fc`](https://github.com/apache/spark/commit/dfc59fc7426eedf40b66d63f75d6f3f133ec240d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14471: [SPARK-14387][SQL] Enable Hive-1.x ORC compatibility wit...

2017-10-21 Thread dongjoon-hyun

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/14471
  
Hi, @rajeshbalamohan .
SPARK-14387 is resolved now. Could you close this PR?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19538: [SPARK-20393][WEBU UI][BACKPORT-2.0] Strengthen S...

2017-10-21 Thread ambauma

Github user ambauma commented on a diff in the pull request:

https://github.com/apache/spark/pull/19538#discussion_r146111073
  
--- Diff: core/src/main/scala/org/apache/spark/ui/UIUtils.scala ---
@@ -506,4 +510,33 @@ private[spark] object UIUtils extends Logging {
 
   def getTimeZoneOffset() : Int =
 TimeZone.getDefault().getOffset(System.currentTimeMillis()) / 1000 / 60
+
+  /**
+  * Return the correct Href after checking if master is running in the
+  * reverse proxy mode or not.
+  */
+  def makeHref(proxy: Boolean, id: String, origHref: String): String = {
--- End diff --

I think this method came with the original patch.  I don't see anything 
calling it.  I will remove it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19524: [SPARK-22302][INFRA] Remove manual backports for ...

2017-10-21 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19524


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19524: [SPARK-22302][INFRA] Remove manual backports for subproc...

2017-10-21 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19524
  
Merged to master.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19546: [SPARK-21055][SQL][FOLLOW-UP] replace grouping__i...

2017-10-21 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19546


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19547: [SPARK-20331][SQL][FOLLOW-UP] Add a SQLConf for e...

2017-10-21 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19547


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19546: [SPARK-21055][SQL][FOLLOW-UP] replace grouping__id with ...

2017-10-21 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/19546
  
Thanks! Merged to master.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19546: [SPARK-21055][SQL][FOLLOW-UP] replace grouping__i...

2017-10-21 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/19546#discussion_r146110250
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 
---
@@ -1376,223 +1376,125 @@ class SQLQuerySuite extends QueryTest with 
SQLTestUtils with TestHiveSingleton {
   }
 
   test("SPARK-8976 Wrong Result for Rollup #1") {
--- End diff --

That is fine. We do not have such a rule.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19547: [SPARK-20331][SQL][FOLLOW-UP] Add a SQLConf for enhanced...

2017-10-21 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/19547
  
Thanks! Merged to master.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19547: [SPARK-20331][SQL][FOLLOW-UP] Add a SQLConf for e...

2017-10-21 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/19547#discussion_r146110223
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -585,6 +585,35 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
* Unsupported predicates are skipped.
*/
   def convertFilters(table: Table, filters: Seq[Expression]): String = {
+if (SQLConf.get.advancedPartitionPredicatePushdownEnabled) {
+  convertComplexFilters(table, filters)
+} else {
+  convertBasicFilters(table, filters)
+}
--- End diff --

Thanks! We can do it when enhancing `convertComplexFilters `. Here, it is 
just to keep the original codes (from Spark 2.2) untouched.  


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19547: [SPARK-20331][SQL][FOLLOW-UP] Add a SQLConf for e...

2017-10-21 Thread dongjoon-hyun

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/19547#discussion_r146109395
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -585,6 +585,35 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
* Unsupported predicates are skipped.
*/
   def convertFilters(table: Table, filters: Seq[Expression]): String = {
+if (SQLConf.get.advancedPartitionPredicatePushdownEnabled) {
+  convertComplexFilters(table, filters)
+} else {
+  convertBasicFilters(table, filters)
+}
--- End diff --

Nit, Can we remove the duplication of `varcharKeys` logic by moving into 
`convertFilters` like the following?
```scala
  def convertFilters(table: Table, filters: Seq[Expression]): String = {
// hive varchar is treated as catalyst string, but hive varchar can't 
be pushed down.
lazy val varcharKeys = table.getPartitionKeys.asScala
  .filter(col => 
col.getType.startsWith(serdeConstants.VARCHAR_TYPE_NAME) ||
col.getType.startsWith(serdeConstants.CHAR_TYPE_NAME))
  .map(col => col.getName).toSet

if (SQLConf.get.advancedPartitionPredicatePushdownEnabled) {
  convertComplexFilters(table, filters, varcharKeys)
} else {
  convertBasicFilters(table, filters, varcharKeys)
}
  }
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19546: [SPARK-21055][SQL][FOLLOW-UP] replace grouping__i...

2017-10-21 Thread dongjoon-hyun

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/19546#discussion_r146108930
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 
---
@@ -1376,223 +1376,125 @@ class SQLQuerySuite extends QueryTest with 
SQLTestUtils with TestHiveSingleton {
   }
 
   test("SPARK-8976 Wrong Result for Rollup #1") {
--- End diff --

Nit, do we need to keep `SPARK-21055` in the test case name here, too?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19545: [SPARK-21929][SQL] Support `ALTER TABLE table_name ADD C...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19545
  
**[Test build #82949 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82949/testReport)**
 for PR 19545 at commit 
[`dfc59fc`](https://github.com/apache/spark/commit/dfc59fc7426eedf40b66d63f75d6f3f133ec240d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19545: [SPARK-21929][SQL] Support `ALTER TABLE table_name ADD C...

2017-10-21 Thread dongjoon-hyun

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/19545
  
Retest this please.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19514: [SPARK-21551][Python] Increase timeout for PythonRDD.ser...

2017-10-21 Thread srowen

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19514
  
Oops yes meant 2.0


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...

2017-10-21 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19535
  
Thanks @felixcheung. Will update soon.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19514: [SPARK-21551][Python] Increase timeout for PythonRDD.ser...

2017-10-21 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19514
  
@srowen, not a big deal but .. I believe it's merged to 2.0 instead of 
master though.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19506: [SPARK-22285] [SQL] Change implementation of ApproxCount...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19506
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82948/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19506: [SPARK-22285] [SQL] Change implementation of ApproxCount...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19506
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19506: [SPARK-22285] [SQL] Change implementation of ApproxCount...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19506
  
**[Test build #82948 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82948/testReport)**
 for PR 19506 at commit 
[`49b4ac2`](https://github.com/apache/spark/commit/49b4ac2ae6bd53dd529a5f1f1ea55fe5fd987192).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19506: [SPARK-22285] [SQL] Change implementation of ApproxCount...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19506
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19506: [SPARK-22285] [SQL] Change implementation of ApproxCount...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19506
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82947/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19506: [SPARK-22285] [SQL] Change implementation of ApproxCount...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19506
  
**[Test build #82947 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82947/testReport)**
 for PR 19506 at commit 
[`1b75428`](https://github.com/apache/spark/commit/1b754284ddba8ebe0e551580b55a8b907607c30a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19543: [SPARK-19606][MESOS] Support constraints in spark-dispat...

2017-10-21 Thread pmackles

Github user pmackles commented on the issue:

https://github.com/apache/spark/pull/19543
  
My apologies. Reread the docs and ran style checks locally this time. 
Fingers crossed


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19547: [SPARK-20331][SQL][FOLLOW-UP] Add a SQLConf for enhanced...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19547
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82944/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19547: [SPARK-20331][SQL][FOLLOW-UP] Add a SQLConf for enhanced...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19547
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19547: [SPARK-20331][SQL][FOLLOW-UP] Add a SQLConf for enhanced...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19547
  
**[Test build #82944 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82944/testReport)**
 for PR 19547 at commit 
[`4057602`](https://github.com/apache/spark/commit/40576020301e551dc0c66375b00a5fcc150e9ab7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19546: [SPARK-21055][SQL][FOLLOW-UP] replace grouping__id with ...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19546
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82945/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19546: [SPARK-21055][SQL][FOLLOW-UP] replace grouping__id with ...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19546
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19546: [SPARK-21055][SQL][FOLLOW-UP] replace grouping__id with ...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19546
  
**[Test build #82945 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82945/testReport)**
 for PR 19546 at commit 
[`35b07e2`](https://github.com/apache/spark/commit/35b07e2edc185213b1582d15e211e0d808c72333).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19506: [SPARK-22285] [SQL] Change implementation of ApproxCount...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19506
  
**[Test build #82948 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82948/testReport)**
 for PR 19506 at commit 
[`49b4ac2`](https://github.com/apache/spark/commit/49b4ac2ae6bd53dd529a5f1f1ea55fe5fd987192).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19514: [SPARK-21551][Python] Increase timeout for PythonRDD.ser...

2017-10-21 Thread srowen

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19514
  
Merged to master @FRosner ; you'll have to close this manually too


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19506: [SPARK-22285] [SQL] Change implementation of Appr...

2017-10-21 Thread wzhfy

Github user wzhfy commented on a diff in the pull request:

https://github.com/apache/spark/pull/19506#discussion_r146100286
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxCountDistinctForIntervals.scala
 ---
@@ -239,4 +221,23 @@ case class ApproxCountDistinctForIntervals(
   override def dataType: DataType = ArrayType(LongType)
 
   override def prettyName: String = "approx_count_distinct_for_intervals"
+
+  override def serialize(obj: Array[Long]): Array[Byte] = {
+val buffer = ByteBuffer.wrap(new Array(obj.length * Longs.BYTES))
--- End diff --

Changed to unsafe writing, could you take another look?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19506: [SPARK-22285] [SQL] Change implementation of ApproxCount...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19506
  
**[Test build #82947 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82947/testReport)**
 for PR 19506 at commit 
[`1b75428`](https://github.com/apache/spark/commit/1b754284ddba8ebe0e551580b55a8b907607c30a).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19543: [SPARK-19606][MESOS] Support constraints in spark-dispat...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19543
  
**[Test build #82946 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82946/testReport)**
 for PR 19543 at commit 
[`efdc09c`](https://github.com/apache/spark/commit/efdc09c20aaffe56d7b214f2e54db39afd66374f).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19543: [SPARK-19606][MESOS] Support constraints in spark-dispat...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19543
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19543: [SPARK-19606][MESOS] Support constraints in spark-dispat...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19543
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82946/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19547: [SPARK-20331][SQL][FOLLOW-UP] Add a SQLConf for enhanced...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19547
  
**[Test build #82944 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82944/testReport)**
 for PR 19547 at commit 
[`4057602`](https://github.com/apache/spark/commit/40576020301e551dc0c66375b00a5fcc150e9ab7).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19543: [SPARK-19606][MESOS] Support constraints in spark-dispat...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19543
  
**[Test build #82946 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82946/testReport)**
 for PR 19543 at commit 
[`efdc09c`](https://github.com/apache/spark/commit/efdc09c20aaffe56d7b214f2e54db39afd66374f).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19546: [SPARK-21055][SQL][FOLLOW-UP] replace grouping__id with ...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19546
  
**[Test build #82945 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82945/testReport)**
 for PR 19546 at commit 
[`35b07e2`](https://github.com/apache/spark/commit/35b07e2edc185213b1582d15e211e0d808c72333).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19546: [SPARK-21055][SQL][FOLLOW-UP] replace grouping__id with ...

2017-10-21 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/19546
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19547: [SPARK-20331][SQL][FOLLOW-UP] Add a SQLConf for e...

2017-10-21 Thread gatorsmile

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/19547

[SPARK-20331][SQL][FOLLOW-UP] Add a SQLConf for enhanced Hive partition 
pruning predicate pushdown 

## What changes were proposed in this pull request?
This is a follow-up PR of https://github.com/apache/spark/pull/17633. 

This PR is to add a conf 
`spark.sql.hive.advancedPartitionPredicatePushdown.enabled`, which can be used 
to turn the enhancement off.

## How was this patch tested?
Add a test case

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark Spark20331FollowUp

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19547.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19547


commit 40576020301e551dc0c66375b00a5fcc150e9ab7
Author: gatorsmile 
Date:   2017-10-21T06:56:30Z

fix.




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19543: [SPARK-19606][MESOS] Support constraints in spark-dispat...

2017-10-21 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/19543
  
Jenkins, retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19528: [SPARK-20393][WEBU UI][1.6] Strengthen Spark to p...

2017-10-21 Thread felixcheung

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/19528#discussion_r146098569
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -22,6 +22,7 @@ import javax.servlet.http.HttpServletRequest
 
 import scala.collection.mutable.{HashMap, ListBuffer}
 import scala.xml._
+import scala.collection.JavaConverters._
--- End diff --

is this needed?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19272: [Spark-21842][Mesos] Support Kerberos ticket renewal and...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19272
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19272: [Spark-21842][Mesos] Support Kerberos ticket renewal and...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19272
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82941/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19545: [SPARK-21929][SQL] Support `ALTER TABLE table_name ADD C...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19545
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82942/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19546: [SPARK-21055][SQL][FOLLOW-UP] replace grouping__id with ...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19546
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82943/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19545: [SPARK-21929][SQL] Support `ALTER TABLE table_name ADD C...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19545
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19546: [SPARK-21055][SQL][FOLLOW-UP] replace grouping__id with ...

2017-10-21 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19546
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19545: [SPARK-21929][SQL] Support `ALTER TABLE table_name ADD C...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19545
  
**[Test build #82942 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82942/testReport)**
 for PR 19545 at commit 
[`dfc59fc`](https://github.com/apache/spark/commit/dfc59fc7426eedf40b66d63f75d6f3f133ec240d).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19546: [SPARK-21055][SQL][FOLLOW-UP] replace grouping__id with ...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19546
  
**[Test build #82943 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82943/testReport)**
 for PR 19546 at commit 
[`35b07e2`](https://github.com/apache/spark/commit/35b07e2edc185213b1582d15e211e0d808c72333).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19272: [Spark-21842][Mesos] Support Kerberos ticket renewal and...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19272
  
**[Test build #82941 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82941/testReport)**
 for PR 19272 at commit 
[`c95f80b`](https://github.com/apache/spark/commit/c95f80b23d47ea4640cea2b4a185fa4bf9e9f33d).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19538: [SPARK-20393][WEBU UI][BACKPORT-2.0] Strengthen S...

2017-10-21 Thread felixcheung

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/19538#discussion_r146098560
  
--- Diff: core/src/main/scala/org/apache/spark/ui/UIUtils.scala ---
@@ -506,4 +510,33 @@ private[spark] object UIUtils extends Logging {
 
   def getTimeZoneOffset() : Int =
 TimeZone.getDefault().getOffset(System.currentTimeMillis()) / 1000 / 60
+
+  /**
+  * Return the correct Href after checking if master is running in the
+  * reverse proxy mode or not.
+  */
+  def makeHref(proxy: Boolean, id: String, origHref: String): String = {
--- End diff --

this is not in the 2.0 PR?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19546: [SPARK-21055][SQL][FOLLOW-UP] replace grouping__id with ...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19546
  
**[Test build #82943 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82943/testReport)**
 for PR 19546 at commit 
[`35b07e2`](https://github.com/apache/spark/commit/35b07e2edc185213b1582d15e211e0d808c72333).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19546: [SPARK-21055][SQL][FOLLOW-UP] replace grouping__id with ...

2017-10-21 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/19546
  
cc @jiangxb1987 @cloud-fan 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19546: [SPARK-21055][SQL][FOLLOW-UP] replace grouping__i...

2017-10-21 Thread gatorsmile

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/19546

[SPARK-21055][SQL][FOLLOW-UP] replace grouping__id with grouping_id()

## What changes were proposed in this pull request?
Simplifies the test cases that were added in the PR 
https://github.com/apache/spark/pull/18270.

## How was this patch tested?
N/A

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark backportSPARK-21055

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19546.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19546


commit 35b07e2edc185213b1582d15e211e0d808c72333
Author: gatorsmile 
Date:   2017-10-21T06:30:23Z

fix.




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19545: [SPARK-21929][SQL] Support `ALTER TABLE table_name ADD C...

2017-10-21 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19545
  
**[Test build #82942 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82942/testReport)**
 for PR 19545 at commit 
[`dfc59fc`](https://github.com/apache/spark/commit/dfc59fc7426eedf40b66d63f75d6f3f133ec240d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19545: [SPARK-21929][SQL] Support `ALTER TABLE table_name ADD C...

2017-10-21 Thread dongjoon-hyun

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/19545
  
Thank you for review, @gatorsmile ! I updated the PR.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

98 matches

Mail list logo