[GitHub] spark issue #18103: [SPARK-20876][SQL]If the input parameter is float type f...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18103
  
**[Test build #77450 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77450/testReport)**
 for PR 18103 at commit 
[`660c038`](https://github.com/apache/spark/commit/660c038a5b227454abb13a392c9ba4156c4e6ba2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18127: [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException whe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18127
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77449/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18127: [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException whe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18127
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18127: [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException whe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18127
  
**[Test build #77449 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77449/testReport)**
 for PR 18127 at commit 
[`6a622b0`](https://github.com/apache/spark/commit/6a622b071fdf2e86e1849a4473cf9525e2ae3de0).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18016: [SPARK-20786][SQL]Improve ceil and floor handle the valu...

[heary-cao]

Github user heary-cao commented on the issue:

https://github.com/apache/spark/pull/18016
  
@ueshin ,
thank you for your suggestion. 
modify `doGenCode() `pr: #18103


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18127: [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException whe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18127
  
**[Test build #77449 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77449/testReport)**
 for PR 18127 at commit 
[`6a622b0`](https://github.com/apache/spark/commit/6a622b071fdf2e86e1849a4473cf9525e2ae3de0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18127: [SPARK-6628][SQL][Branch-2.1] Fix ClassCastExcept...

[weiqingy]

GitHub user weiqingy opened a pull request:

https://github.com/apache/spark/pull/18127

[SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql 
statement 'insert into' on hbase table

## What changes were proposed in this pull request?

The issue of SPARK-6628 is:
```
org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat 
```
cannot be cast to
```
org.apache.hadoop.hive.ql.io.HiveOutputFormat
```
The reason is:
```
public interface HiveOutputFormat extends OutputFormat {…}

public class HiveHBaseTableOutputFormat extends
TableOutputFormat implements
OutputFormat {...}
```
From the two snippets above, we can see both `HiveHBaseTableOutputFormat` 
and `HiveOutputFormat` `extends`/`implements` `OutputFormat`, and can not cast 
to each other. 

For Spark 1.6, 2.0, 2.1, Spark initials the `outputFormat` in 
`SparkHiveWriterContainer`. For Spark 2.2+,  Spark initials the `outputFormat` 
in `HiveFileFormat`.
```
@transient private lazy val outputFormat =
jobConf.value.getOutputFormat.asInstanceOf[HiveOutputFormat[AnyRef, 
Writable]]
```
`outputFormat` above has to be  `HiveOutputFormat`. However, when users 
insert data into hbase, the outputFormat is `HiveHBaseTableOutputFormat`, it 
isn't instance of `HiveOutputFormat`.

This PR is to make `outputFormat` to be "null" when the `OutputFormat` is 
not an instance of `HiveOutputFormat`. This change should be safe since 
`outputFormat` is only used to get the file extension in function 
[`getFileExtension()`](https://github.com/apache/spark/blob/branch-2.1/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveWriterContainers.scala#L101).
 

We can also submit this PR to Master branch.

## How was this patch tested?
Manually test.
(1) create a HBase table with Hive:
```
CREATE TABLE testwq100 (row_key string COMMENT 'from deserializer', 
application string COMMENT 'from deserializer', starttime timestamp COMMENT 
'from deserializer', endtime timestamp COMMENT 'from deserializer', status 
string COMMENT 'from deserializer', statusid smallint COMMENT 'from 
deserializer',   insertdate timestamp COMMENT 'from deserializer', count int 
COMMENT 'from deserializer', errordesc string COMMENT 'from deserializer') ROW 
FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY 
'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( 
'hbase.columns.mapping'='cf1:application,cf1:starttime,cf1:endtime,cf1:Status,cf1:StatusId,cf1:InsertDate,cf1:count,cf1:ErrorDesc',
 'line.delim'='\\n',   'mapkey.delim'='\\u0003', 
'serialization.format'='\\u0001') TBLPROPERTIES 
('transient_lastDdlTime'='1489696241', 'hbase.table.name' = 'xyz', 
'hbase.mapred.output.outputtable' = 'xyz')
```
(2) verify:

**Before:**

Insert data into the Hbase table `testwq100` from Spark SQL:
```
scala> sql(s"INSERT INTO testwq100 VALUES 
('AA1M22','AA1M122','2011722','201156','Starte1d6',45,20,1,'ad1')")
17/05/26 00:09:10 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 3)
java.lang.ClassCastException: 
org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat cannot be cast to 
org.apache.hadoop.hive.ql.io.HiveOutputFormat
at 
org.apache.spark.sql.hive.SparkHiveWriterContainer.outputFormat$lzycompute(hiveWriterContainers.scala:82)
at 
org.apache.spark.sql.hive.SparkHiveWriterContainer.outputFormat(hiveWriterContainers.scala:81)
at 
org.apache.spark.sql.hive.SparkHiveWriterContainer.getOutputName(hiveWriterContainers.scala:101)
at 
org.apache.spark.sql.hive.SparkHiveWriterContainer.initWriters(hiveWriterContainers.scala:125)
at 
org.apache.spark.sql.hive.SparkHiveWriterContainer.executorSideSetup(hiveWriterContainers.scala:94)
at 
org.apache.spark.sql.hive.SparkHiveWriterContainer.writeToFile(hiveWriterContainers.scala:182)
at 
org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:210)
at 
org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:210)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
17/05/26 00:09:10 WARN TaskSetManager: Lost task 0.0 in stage 3.0 (TID 3, 
localhost, executor driver): java.lang.ClassCastException: 
org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat cannot be cast to 

[GitHub] spark pull request #18126: [SPARK-20843][Core]Add a config to set driver ter...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18126


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18126: [SPARK-20843][Core]Add a config to set driver terminate ...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/18126
  
Thanks! Merging to master and 2.2.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

[kevinyu98]

Github user kevinyu98 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12646#discussion_r118813943
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 
---
@@ -2015,4 +2015,121 @@ class SQLQuerySuite extends QueryTest with 
SQLTestUtils with TestHiveSingleton {
   checkAnswer(table.filter($"p" === "p1\" and q=\"q1").select($"a"), 
Row(4))
 }
   }
+
+  test("TRIM function-BOTH") {
--- End diff --

sure, I will remove the test case from hive/execution/SQLQuerySuite.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

[kevinyu98]

Github user kevinyu98 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12646#discussion_r118813930
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -2616,4 +2617,66 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
 val e = intercept[AnalysisException](sql("SELECT nvl(1, 2, 3)"))
 assert(e.message.contains("Invalid number of arguments"))
   }
+
+  test("TRIM function-BOTH") {
+val ae1 = intercept[ParseException]{
+  sql("select lower(BOTH 'S' FROM 'SS abc S')").head
--- End diff --

sure


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15831: [SPARK-18385][ML] Make the transformer's natively in ml ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15831
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15831: [SPARK-18385][ML] Make the transformer's natively in ml ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15831
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77448/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15831: [SPARK-18385][ML] Make the transformer's natively in ml ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15831
  
**[Test build #77448 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77448/testReport)**
 for PR 15831 at commit 
[`89e6858`](https://github.com/apache/spark/commit/89e6858545d5f9b064b590b3e3e5f34bcb3bfa82).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

[wzhfy]

Github user wzhfy commented on a diff in the pull request:

https://github.com/apache/spark/pull/12646#discussion_r118813705
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 
---
@@ -2015,4 +2015,121 @@ class SQLQuerySuite extends QueryTest with 
SQLTestUtils with TestHiveSingleton {
   checkAnswer(table.filter($"p" === "p1\" and q=\"q1").select($"a"), 
Row(4))
 }
   }
+
+  test("TRIM function-BOTH") {
--- End diff --

Usually we add tests here when we need to test something with hive support.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

[wzhfy]

Github user wzhfy commented on a diff in the pull request:

https://github.com/apache/spark/pull/12646#discussion_r118813687
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 
---
@@ -2015,4 +2015,121 @@ class SQLQuerySuite extends QueryTest with 
SQLTestUtils with TestHiveSingleton {
   checkAnswer(table.filter($"p" === "p1\" and q=\"q1").select($"a"), 
Row(4))
 }
   }
+
+  test("TRIM function-BOTH") {
--- End diff --

These tests can also be put in sql/SQLQuerySuite. By the way, IIUC, the 
codeGen path is already tested in `StringExpressionSuite` by `checkEvaluation`, 
right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

[kevinyu98]

Github user kevinyu98 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12646#discussion_r118813536
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -2616,4 +2617,66 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
 val e = intercept[AnalysisException](sql("SELECT nvl(1, 2, 3)"))
 assert(e.message.contains("Invalid number of arguments"))
   }
+
+  test("TRIM function-BOTH") {
--- End diff --

sure


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18103: [SPARK-20876][SQL]If the input parameter is float type f...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18103
  
LGTM except one comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18103: [SPARK-20876][SQL]If the input parameter is float type f...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18103
  
Could you remove the sql statements from opterators.sql after you moving 
them to mathExpressionSuite?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

[kevinyu98]

Github user kevinyu98 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12646#discussion_r118813507
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 
---
@@ -2015,4 +2015,121 @@ class SQLQuerySuite extends QueryTest with 
SQLTestUtils with TestHiveSingleton {
   checkAnswer(table.filter($"p" === "p1\" and q=\"q1").select($"a"), 
Row(4))
 }
   }
+
+  test("TRIM function-BOTH") {
--- End diff --

what do you think?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

[kevinyu98]

Github user kevinyu98 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12646#discussion_r118813501
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 
---
@@ -2015,4 +2015,121 @@ class SQLQuerySuite extends QueryTest with 
SQLTestUtils with TestHiveSingleton {
   checkAnswer(table.filter($"p" === "p1\" and q=\"q1").select($"a"), 
Row(4))
 }
   }
+
+  test("TRIM function-BOTH") {
--- End diff --

For hive/execution/SQLQuerySuite, it will test the codeGen path 
(doGenCode). The source string is from column name.
 But I can remove those test cases not go through the codeGen path, both 
source string and trim string are string constant. 

`checkAnswer(sql("SELECT TRIM(BOTH '数$,.' FROM '数.数,数%') from 
trimBoth"), Row("%")) checkAnswer(sql("SELECT TRIM(BOTH '敠数' FROM '敠敠
数敠敠') from trimBoth"), Row("")) checkAnswer(sql("SELECT TRIM(BOTH '搠
abc' FROM 'a数b') from trimBoth"), Row("数"))`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18103: [SPARK-20876][SQL]If the input parameter is float type f...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18103
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18103: [SPARK-20876][SQL]If the input parameter is float type f...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18103
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77444/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18103: [SPARK-20876][SQL]If the input parameter is float type f...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18103
  
**[Test build #77444 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77444/testReport)**
 for PR 18103 at commit 
[`4cdb44e`](https://github.com/apache/spark/commit/4cdb44ef115b680893db64aacce4def32ccec690).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18114: [SPARK-20889][SparkR] Grouped documentation for DATETIME...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18114
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18114: [SPARK-20889][SparkR] Grouped documentation for DATETIME...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18114
  
**[Test build #77447 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77447/testReport)**
 for PR 18114 at commit 
[`944aa92`](https://github.com/apache/spark/commit/944aa92e4c35015010ee9cdaf913f5639670acb6).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `#' @param x if of class Column, it is used to perform arithmatic 
operations with


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18114: [SPARK-20889][SparkR] Grouped documentation for DATETIME...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18114
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77447/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17989: [SPARK-6628][SQL][Branch-1.6] Fix ClassCastExcept...

[weiqingy]

Github user weiqingy closed the pull request at:

https://github.com/apache/spark/pull/17989


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17989: [SPARK-6628][SQL][Branch-1.6] Fix ClassCastException whe...

[weiqingy]

Github user weiqingy commented on the issue:

https://github.com/apache/spark/pull/17989
  
Spark 1.6 is a little old. I'll move the change to branch-2.1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15831: [SPARK-18385][ML] Make the transformer's natively in ml ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15831
  
**[Test build #77448 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77448/testReport)**
 for PR 15831 at commit 
[`89e6858`](https://github.com/apache/spark/commit/89e6858545d5f9b064b590b3e3e5f34bcb3bfa82).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18019: [SPARK-20748][SQL] Add built-in SQL function CH[A...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18019


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18019: [SPARK-20748][SQL] Add built-in SQL function CH[A]R.

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18019
  
Thanks! Merging to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18119: [BACKPORT-2.2][SPARK-19372][SQL] Fix throwing a J...

[kiszk]

Github user kiszk closed the pull request at:

https://github.com/apache/spark/pull/18119


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18119: [BACKPORT-2.2][SPARK-19372][SQL] Fix throwing a Java exc...

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/18119
  
@zsxwing , sure, I submit another PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18114: [SPARK-20889][SparkR] Grouped documentation for DATETIME...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18114
  
**[Test build #77447 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77447/testReport)**
 for PR 18114 at commit 
[`944aa92`](https://github.com/apache/spark/commit/944aa92e4c35015010ee9cdaf913f5639670acb6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18114: [SPARK-20889][SparkR] Grouped documentation for DATETIME...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/18114
  
Jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17113: [SPARK-13669][Core] Improve the blacklist mechanism to h...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17113
  
**[Test build #77446 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77446/testReport)**
 for PR 17113 at commit 
[`524fbfc`](https://github.com/apache/spark/commit/524fbfcdd661f5919515b8e18515d346003a3ebe).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18104: [SPARK-20877][SPARKR][WIP] add timestamps to test...

[felixcheung]

GitHub user felixcheung reopened a pull request:

https://github.com/apache/spark/pull/18104

[SPARK-20877][SPARKR][WIP] add timestamps to test runs

## What changes were proposed in this pull request?

to investigate how long they run

## How was this patch tested?

Jenkins, AppVeyor

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark rtimetest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18104.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18104


commit dab72a60441e337e9143c4144795a723d5cc0867
Author: Felix Cheung 
Date:   2017-05-25T03:53:24Z

timestamp tests

commit 313dcbc99c408c81d6bd5e5395bb373e1d0f418a
Author: Felix Cheung 
Date:   2017-05-25T04:46:25Z

set timezone

commit a72ab8c3153743ee5b5d0fe4ba797023aac6e88c
Author: Felix Cheung 
Date:   2017-05-25T05:00:25Z

fix

commit 3273a0d7f5ef35cf2cfc0330f06c4f78922ed86f
Author: Felix Cheung 
Date:   2017-05-26T03:46:41Z

update timer, --as-cran as a test

commit ce1c99070c3a320f1029d8b735a684883a09362a
Author: Felix Cheung 
Date:   2017-05-26T04:03:14Z

typos

commit 1d18d7b808a61ef6e32a7caa46427857e2c12b85
Author: Felix Cheung 
Date:   2017-05-26T04:03:29Z

one more

commit 21ca8f0d2928162fa4a05588465f57241dcd8132
Author: Felix Cheung 
Date:   2017-05-26T04:50:17Z

skip tests as cran




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18104: [SPARK-20877][SPARKR][WIP] add timestamps to test...

[felixcheung]

Github user felixcheung closed the pull request at:

https://github.com/apache/spark/pull/18104


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17113: [SPARK-13669][Core] Improve the blacklist mechanism to h...

[jerryshao]

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/17113
  
Sorry @tgravescs I didn't test executor killing in real cluster. There has 
bug in it, so I pushed a commit to fix it. Thanks for your reviewing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18121: [SPARK-20897][SQL] cached self-join should not fail

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18121
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77445/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18121: [SPARK-20897][SQL] cached self-join should not fail

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18121
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18121: [SPARK-20897][SQL] cached self-join should not fail

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18121
  
**[Test build #77445 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77445/testReport)**
 for PR 18121 at commit 
[`e91311c`](https://github.com/apache/spark/commit/e91311cceb8cc25d752eb6174e020c050291ab57).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18121: [SPARK-20897][SQL] cached self-join should not fail

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18121
  
**[Test build #77445 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77445/testReport)**
 for PR 18121 at commit 
[`e91311c`](https://github.com/apache/spark/commit/e91311cceb8cc25d752eb6174e020c050291ab57).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18121: [SPARK-20897][SQL] cached self-join should not fa...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/18121#discussion_r118812139
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/Exchange.scala 
---
@@ -58,6 +59,10 @@ case class ReusedExchangeExec(override val output: 
Seq[Attribute], child: Exchan
   override protected[sql] def doExecuteBroadcast[T](): 
broadcast.Broadcast[T] = {
 child.executeBroadcast()
   }
+
+  override def outputPartitioning: Partitioning = child.outputPartitioning
--- End diff --

good catch!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18082: [SPARK-20665][SQL][FOLLOW-UP]Move test case to Ma...

[10110346]

Github user 10110346 closed the pull request at:

https://github.com/apache/spark/pull/18082


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18023: [SPARK-12139] [SQL] REGEX Column Specification

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18023
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77443/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18103: [SPARK-20876][SQL]If the input parameter is float...

[10110346]

Github user 10110346 commented on a diff in the pull request:

https://github.com/apache/spark/pull/18103#discussion_r118812033
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MathExpressionsSuite.scala
 ---
@@ -258,6 +258,16 @@ class MathExpressionsSuite extends SparkFunSuite with 
ExpressionEvalHelper {
 checkConsistencyBetweenInterpretedAndCodegen(Ceil, DecimalType(25, 3))
 checkConsistencyBetweenInterpretedAndCodegen(Ceil, DecimalType(25, 0))
 checkConsistencyBetweenInterpretedAndCodegen(Ceil, DecimalType(5, 0))
+
+val doublePi: Double = 3.1415
+val floatPi: Float = 3.1415f
+val longPi: Long = 31415926535897912L
--- End diff --

I closed the PR #18082  temporarily 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18113: [SPARK-20890][SQL] Added min and max typed aggreg...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/18113#discussion_r118812026
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/typedaggregators.scala
 ---
@@ -38,7 +38,6 @@ class TypedSumDouble[IN](val f: IN => Double) extends 
Aggregator[IN, Double, Dou
 
   // Java api support
   def this(f: MapFunction[IN, java.lang.Double]) = this(x => 
f.call(x).asInstanceOf[Double])
-
--- End diff --

nit: keep the original style (do not delete brank line)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18113: [SPARK-20890][SQL] Added min and max typed aggreg...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/18113#discussion_r118812028
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/typedaggregators.scala
 ---
@@ -56,7 +55,6 @@ class TypedSumLong[IN](val f: IN => Long) extends 
Aggregator[IN, Long, Long] {
 
   // Java api support
   def this(f: MapFunction[IN, java.lang.Long]) = this(x => 
f.call(x).asInstanceOf[Long])
-
--- End diff --

nit: ditto


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18113: [SPARK-20890][SQL] Added min and max typed aggreg...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/18113#discussion_r118812022
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/typedaggregators.scala
 ---
@@ -99,3 +97,67 @@ class TypedAverage[IN](val f: IN => Double) extends 
Aggregator[IN, (Double, Long
 toColumn.asInstanceOf[TypedColumn[IN, java.lang.Double]]
   }
 }
+
+class TypedMinDouble[IN](val f: IN => Double) extends Aggregator[IN, 
Double, Double] {
+  override def zero: Double = Double.PositiveInfinity
+  override def reduce(b: Double, a: IN): Double = if (b < f(a)) b else f(a)
--- End diff --

`math.min(b, f(a))`.  Similar to other places


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18023: [SPARK-12139] [SQL] REGEX Column Specification

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18023
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18023: [SPARK-12139] [SQL] REGEX Column Specification

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18023
  
**[Test build #77443 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77443/testReport)**
 for PR 18023 at commit 
[`a27023c`](https://github.com/apache/spark/commit/a27023c69ace6026a345dc745206c20e71ac4299).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18117: [SPARK-19659][CORE][FOLLOW-UP] Fetch big blocks t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18117


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18117: [SPARK-19659][CORE][FOLLOW-UP] Fetch big blocks to disk ...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18117
  
thanks for the review, merging to master/2.2!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17947: [SPARK-20708][CORE] Make `addExclusionRules` up-to-date

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/17947
  
Thank you, @jiangxb1987 .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18103: [SPARK-20876][SQL]If the input parameter is float type f...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18103
  
**[Test build #77444 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77444/testReport)**
 for PR 18103 at commit 
[`4cdb44e`](https://github.com/apache/spark/commit/4cdb44ef115b680893db64aacce4def32ccec690).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

[wzhfy]

Github user wzhfy commented on a diff in the pull request:

https://github.com/apache/spark/pull/12646#discussion_r118811306
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -2616,4 +2617,66 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
 val e = intercept[AnalysisException](sql("SELECT nvl(1, 2, 3)"))
 assert(e.message.contains("Invalid number of arguments"))
   }
+
+  test("TRIM function-BOTH") {
+val ae1 = intercept[ParseException]{
+  sql("select lower(BOTH 'S' FROM 'SS abc S')").head
--- End diff --

ltrim/rtrim can't using these options, right? We can use them as negative 
test for function names, and use trim for negative tests for option names.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

[wzhfy]

Github user wzhfy commented on a diff in the pull request:

https://github.com/apache/spark/pull/12646#discussion_r118811261
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -2616,4 +2617,66 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
 val e = intercept[AnalysisException](sql("SELECT nvl(1, 2, 3)"))
 assert(e.message.contains("Invalid number of arguments"))
   }
+
+  test("TRIM function-BOTH") {
+val ae1 = intercept[ParseException]{
+  sql("select lower(BOTH 'S' FROM 'SS abc S')").head
--- End diff --

yea that sounds good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18103: [SPARK-20876][SQL]If the input parameter is float...

[10110346]

Github user 10110346 commented on a diff in the pull request:

https://github.com/apache/spark/pull/18103#discussion_r118810938
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MathExpressionsSuite.scala
 ---
@@ -258,6 +258,16 @@ class MathExpressionsSuite extends SparkFunSuite with 
ExpressionEvalHelper {
 checkConsistencyBetweenInterpretedAndCodegen(Ceil, DecimalType(25, 3))
 checkConsistencyBetweenInterpretedAndCodegen(Ceil, DecimalType(25, 0))
 checkConsistencyBetweenInterpretedAndCodegen(Ceil, DecimalType(5, 0))
+
+val doublePi: Double = 3.1415
+val floatPi: Float = 3.1415f
+val longPi: Long = 31415926535897912L
--- End diff --

yes, actually, i have do it as  #18082


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18103: [SPARK-20876][SQL]If the input parameter is float...

[10110346]

Github user 10110346 commented on a diff in the pull request:

https://github.com/apache/spark/pull/18103#discussion_r118810895
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
 ---
@@ -232,12 +232,13 @@ case class Ceil(child: Expression) extends 
UnaryMathExpression(math.ceil, "CEIL"
   }
 
   override def inputTypes: Seq[AbstractDataType] =
-Seq(TypeCollection(LongType, DoubleType, DecimalType))
+Seq(TypeCollection(DoubleType, DecimalType, LongType, FloatType))
--- End diff --

yes, it  will promote it to DoubleType, but must move LongType to end


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18023: [SPARK-12139] [SQL] REGEX Column Specification

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18023
  
**[Test build #77443 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77443/testReport)**
 for PR 18023 at commit 
[`a27023c`](https://github.com/apache/spark/commit/a27023c69ace6026a345dc745206c20e71ac4299).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18124: [SPARK-20900][YARN] Catch IllegalArgumentException throw...

[jerryshao]

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/18124
  
I'm not sure how this could be happened, "SPARK_YARN_STAGING_DIR" is a 
Spark internal environment variable which should be empty, unless you 
deliberately unset it.

In the JIRA you mentioned you directly running `ApplicationMaster`, so that 
you met this issue, I'm not sure why you need to directly call 
`ApplicationMaster`?

BTW seems you submitted a PR against branch 2.1, why not against master 
branch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18103: [SPARK-20876][SQL]If the input parameter is float...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18103#discussion_r118809673
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
 ---
@@ -232,12 +232,13 @@ case class Ceil(child: Expression) extends 
UnaryMathExpression(math.ceil, "CEIL"
   }
 
   override def inputTypes: Seq[AbstractDataType] =
-Seq(TypeCollection(LongType, DoubleType, DecimalType))
+Seq(TypeCollection(DoubleType, DecimalType, LongType, FloatType))
--- End diff --

No need to add `FloatType`, I think


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18102: [SPARK-20875] Spark should print the log when the direct...

[liu-zhaokun]

Github user liu-zhaokun commented on the issue:

https://github.com/apache/spark/pull/18102
  
@HyukjinKwon 
Hello,the test said there is some failure，is it my problems?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17617: [SPARK-20244][Core] Handle incorrect bytesRead me...

[jerryshao]

Github user jerryshao commented on a diff in the pull request:

https://github.com/apache/spark/pull/17617#discussion_r118809133
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -251,7 +251,13 @@ class HadoopRDD[K, V](
 null
 }
   // Register an on-task-completion callback to close the input stream.
-  context.addTaskCompletionListener{ context => closeIfNeeded() }
+  context.addTaskCompletionListener { context =>
+// Update the bytes read before closing is to make sure lingering 
bytesRead statistics in
+// this thread get correctly added.
+updateBytesRead()
--- End diff --

Close can be called in another thread as I remembered, so I added here to 
avoid lingering bytesRead in task running thread (Some bytes can be read when 
creating InputFormat), also it is no harm to call this `updateBytesRead` again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18103: [SPARK-20876][SQL]If the input parameter is float...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18103#discussion_r118809030
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MathExpressionsSuite.scala
 ---
@@ -258,6 +258,16 @@ class MathExpressionsSuite extends SparkFunSuite with 
ExpressionEvalHelper {
 checkConsistencyBetweenInterpretedAndCodegen(Ceil, DecimalType(25, 3))
 checkConsistencyBetweenInterpretedAndCodegen(Ceil, DecimalType(25, 0))
 checkConsistencyBetweenInterpretedAndCodegen(Ceil, DecimalType(5, 0))
+
+val doublePi: Double = 3.1415
+val floatPi: Float = 3.1415f
+val longPi: Long = 31415926535897912L
--- End diff --

Moving the related test cases from operators.sql here. You can hit the bug 
I mentioned above. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18103: [SPARK-20876][SQL]If the input parameter is float...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18103#discussion_r118809005
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
 ---
@@ -348,12 +349,13 @@ case class Floor(child: Expression) extends 
UnaryMathExpression(math.floor, "FLO
   }
 
   override def inputTypes: Seq[AbstractDataType] =
-Seq(TypeCollection(LongType, DoubleType, DecimalType))
+Seq(TypeCollection(DoubleType, DecimalType, LongType, FloatType))
 
   protected override def nullSafeEval(input: Any): Any = child.dataType 
match {
 case LongType => input.asInstanceOf[Long]
+case FloatType => f(input.asInstanceOf[Float]).toLong
 case DoubleType => f(input.asInstanceOf[Double]).toLong
-case DecimalType.Fixed(precision, scale) => 
input.asInstanceOf[Decimal].floor
+case DecimalType.Fixed(_, _) => input.asInstanceOf[Decimal].floor
   }
 
   override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
--- End diff --

You also need to fix this code path. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17617: [SPARK-20244][Core] Handle incorrect bytesRead me...

[jerryshao]

Github user jerryshao commented on a diff in the pull request:

https://github.com/apache/spark/pull/17617#discussion_r118808801
  
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala 
---
@@ -142,14 +143,18 @@ class SparkHadoopUtil extends Logging {
* Returns a function that can be called to find Hadoop FileSystem bytes 
read. If
* getFSBytesReadOnThreadCallback is called from thread r at time t, the 
returned callback will
* return the bytes read on r since t.
-   *
-   * @return None if the required method can't be found.
*/
   private[spark] def getFSBytesReadOnThreadCallback(): () => Long = {
-val threadStats = 
FileSystem.getAllStatistics.asScala.map(_.getThreadStatistics)
-val f = () => threadStats.map(_.getBytesRead).sum
-val baselineBytesRead = f()
-() => f() - baselineBytesRead
+val f = () => 
FileSystem.getAllStatistics.asScala.map(_.getThreadStatistics.getBytesRead).sum
--- End diff --

For the previous code, `threadStats` and `f` function can be executed in 
two threads, so the metrics we got can be wrong.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18015: [SAPRK-20785][WEB-UI][SQL]Spark should provide jump link...

[guoxiaolongzte]

Github user guoxiaolongzte commented on the issue:

https://github.com/apache/spark/pull/18015
  
@srowen Help to merge to master and modify the status about 
https://issues.apache.org/jira/browse/SPARK-20785.Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18126: [SPARK-20843][Core]Add a config to set driver terminate ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18126
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77438/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18126: [SPARK-20843][Core]Add a config to set driver terminate ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18126
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18126: [SPARK-20843][Core]Add a config to set driver terminate ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18126
  
**[Test build #77438 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77438/testReport)**
 for PR 18126 at commit 
[`ca2c9c5`](https://github.com/apache/spark/commit/ca2c9c53040373e82ff350c1b3c77c1512926cec).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18023: [SPARK-12139] [SQL] REGEX Column Specification

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18023
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18023: [SPARK-12139] [SQL] REGEX Column Specification

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18023
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77442/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18023: [SPARK-12139] [SQL] REGEX Column Specification

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18023
  
**[Test build #77442 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77442/testReport)**
 for PR 18023 at commit 
[`779724d`](https://github.com/apache/spark/commit/779724daae609976d6893a4aec7198bbb4f90095).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18023: [SPARK-12139] [SQL] REGEX Column Specification

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18023
  
**[Test build #77442 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77442/testReport)**
 for PR 18023 at commit 
[`779724d`](https://github.com/apache/spark/commit/779724daae609976d6893a4aec7198bbb4f90095).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13440: [SPARK-15699] [ML] Implement a Chi-Squared test statisti...

[erikerlandson]

Github user erikerlandson commented on the issue:

https://github.com/apache/spark/pull/13440
  
Ready for review


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18023: [SPARK-12139] [SQL] REGEX Column Specification

[janewangfb]

Github user janewangfb commented on a diff in the pull request:

https://github.com/apache/spark/pull/18023#discussion_r118806698
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -1188,6 +1188,12 @@ class Dataset[T] private[sql](
   def col(colName: String): Column = colName match {
 case "*" =>
   Column(ResolvedStar(queryExecution.analyzed.output))
+case ParserUtils.escapedIdentifier(columnNameRegex)
--- End diff --

@cloud-fan I might have misunderstood your last comment "I don't think sql 
a-like syntax is really useful here. How about we create a special cased col 
function that takes a regex?". Can you clarify with some examples? Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18126: [SPARK-20843][Core]Add a config to set driver terminate ...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18126
  
No, I couldn't think of anything else without it being too long winded.  I
agree that the `worker` prefix gives enough meaning, plus whomever uses
this should already know the context that it's intended for.

On Fri, May 26, 2017 at 4:37 PM, Shixiong Zhu 
wrote:

> *@zsxwing* commented on this pull request.
> --
>
> In core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
> :
>
> > @@ -57,7 +57,8 @@ private[deploy] class DriverRunner(
>@volatile private[worker] var finalException: Option[Exception] = None
>
>// Timeout to wait for when trying to terminate a driver.
> -  private val DRIVER_TERMINATE_TIMEOUT_MS = 10 * 1000
> +  private val DRIVER_TERMINATE_TIMEOUT_MS =
> +conf.getTimeAsMs("spark.worker.driverTerminateTimeout", "10s")
>
> spark.worker means this is only for Spark workers, so I think it should
> be obvious. Do you have a better config name?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17617: [SPARK-20244][Core] Handle incorrect bytesRead me...

[jiangxb1987]

Github user jiangxb1987 commented on a diff in the pull request:

https://github.com/apache/spark/pull/17617#discussion_r118805573
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -251,7 +251,13 @@ class HadoopRDD[K, V](
 null
 }
   // Register an on-task-completion callback to close the input stream.
-  context.addTaskCompletionListener{ context => closeIfNeeded() }
+  context.addTaskCompletionListener { context =>
+// Update the bytes read before closing is to make sure lingering 
bytesRead statistics in
+// this thread get correctly added.
+updateBytesRead()
--- End diff --

Will this duplicate with what we do in `close()`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17617: [SPARK-20244][Core] Handle incorrect bytesRead me...

[jiangxb1987]

Github user jiangxb1987 commented on a diff in the pull request:

https://github.com/apache/spark/pull/17617#discussion_r118805015
  
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala 
---
@@ -142,14 +143,18 @@ class SparkHadoopUtil extends Logging {
* Returns a function that can be called to find Hadoop FileSystem bytes 
read. If
* getFSBytesReadOnThreadCallback is called from thread r at time t, the 
returned callback will
* return the bytes read on r since t.
-   *
-   * @return None if the required method can't be found.
*/
   private[spark] def getFSBytesReadOnThreadCallback(): () => Long = {
-val threadStats = 
FileSystem.getAllStatistics.asScala.map(_.getThreadStatistics)
-val f = () => threadStats.map(_.getBytesRead).sum
-val baselineBytesRead = f()
-() => f() - baselineBytesRead
+val f = () => 
FileSystem.getAllStatistics.asScala.map(_.getThreadStatistics.getBytesRead).sum
--- End diff --

Why are you changing this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

[kevinyu98]

Github user kevinyu98 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12646#discussion_r118805550
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -2616,4 +2617,66 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
 val e = intercept[AnalysisException](sql("SELECT nvl(1, 2, 3)"))
 assert(e.message.contains("Invalid number of arguments"))
   }
+
+  test("TRIM function-BOTH") {
+val ae1 = intercept[ParseException]{
+  sql("select lower(BOTH 'S' FROM 'SS abc S')").head
--- End diff --

The keyword BOTH/LEADING/TRAILING/FROM can only be used by trim function. I 
was thinking to show there will be throwing exceptions if other functions try 
to use these keywords.  I can change lower --> LTRIM/RTRIM, what do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18126: [SPARK-20843][Core]Add a config to set driver ter...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/18126#discussion_r118805346
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala ---
@@ -57,7 +57,8 @@ private[deploy] class DriverRunner(
   @volatile private[worker] var finalException: Option[Exception] = None
 
   // Timeout to wait for when trying to terminate a driver.
-  private val DRIVER_TERMINATE_TIMEOUT_MS = 10 * 1000
+  private val DRIVER_TERMINATE_TIMEOUT_MS =
+conf.getTimeAsMs("spark.worker.driverTerminateTimeout", "10s")
--- End diff --

`spark.worker` means this is only for Spark workers, so I think it should 
be obvious. Do you have a better config name?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18023: [SPARK-12139] [SQL] REGEX Column Specification

[janewangfb]

Github user janewangfb commented on a diff in the pull request:

https://github.com/apache/spark/pull/18023#discussion_r118805375
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
 ---
@@ -1230,25 +1230,37 @@ class AstBuilder(conf: SQLConf) extends 
SqlBaseBaseVisitor[AnyRef] with Logging
   }
 
   /**
-   * Create a dereference expression. The return type depends on the type 
of the parent, this can
-   * either be a [[UnresolvedAttribute]] (if the parent is an 
[[UnresolvedAttribute]]), or an
-   * [[UnresolvedExtractValue]] if the parent is some expression.
+   * Create a dereference expression. The return type depends on the type 
of the parent.
+   * If the parent is an [[UnresolvedAttribute]], it can be a 
[[UnresolvedAttribute]] or
+   * a [[UnresolvedRegex]] for regex quoted in ``; if the parent is some 
other expression,
+   * it can be [[UnresolvedExtractValue]].
*/
   override def visitDereference(ctx: DereferenceContext): Expression = 
withOrigin(ctx) {
 val attr = ctx.fieldName.getText
 expression(ctx.base) match {
-  case UnresolvedAttribute(nameParts) =>
-UnresolvedAttribute(nameParts :+ attr)
+  case unresolved_attr @ UnresolvedAttribute(nameParts) =>
+ctx.fieldName.getStart.getText match {
+  case escapedIdentifier(columnNameRegex) if 
conf.supportQuotedRegexColumnName =>
+UnresolvedRegex(columnNameRegex, Some(unresolved_attr.name))
+  case _ =>
+UnresolvedAttribute(nameParts :+ attr)
+}
   case e =>
 UnresolvedExtractValue(e, Literal(attr))
 }
   }
 
   /**
-   * Create an [[UnresolvedAttribute]] expression.
+   * Create an [[UnresolvedAttribute]] expression or a [[UnresolvedRegex]] 
if it is a regex
--- End diff --

@cloud-fan, I updated the code such that for the column field part, always 
use regex if supportQuotedRegexColumnName


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18126: [SPARK-20843][Core]Add a config to set driver terminate ...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/18126
  
> 10s is pretty short for a driver timeout

This is usually not a problem. If worker is trying to kill a driver, it 
often means the driver is unhealthy or being killed by the user intentionally. 
10 seconds to allow shutdown hooks cleaning up resources such as deleting temp 
files is usually enough. Given that replying on shutdown hooks to persist data 
is not common, and we have a configuration for special cases, I think it's fine.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18078: [SPARK-10643] [Core] Make spark-submit download remote f...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18078
  
Thanks! Merging to master/2.1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18078: [SPARK-10643] [Core] Make spark-submit download r...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18078


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18078: [SPARK-10643] [Core] Make spark-submit download remote f...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18078
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18078: [SPARK-10643] [Core] Make spark-submit download remote f...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18078
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77435/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

[kevinyu98]

Github user kevinyu98 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12646#discussion_r118804681
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
 ---
@@ -461,68 +462,269 @@ case class FindInSet(left: Expression, right: 
Expression) extends BinaryExpressi
 }
 
 /**
- * A function that trim the spaces from both ends for the specified string.
- */
+ * A function that takes a character string, removes the leading and/or 
trailing characters matching with the characters
+ * in the trim string, returns the new string. If LEADING/TRAILING/BOTH 
and trimStr keywords are not specified, it
+ * defaults to remove space character from both ends.
+ * trimStr: A character string to be trimmed from the source string, if it 
has multiple characters, the function
+ * searches for each character in the source string, removes the 
characters from the source string until it
+ * encounters the first non-match character.
+ * LEADING: removes any characters from the left end of the source string 
that matches characters in the trim string.
+ * TRAILING: removes any characters from the right end of the source 
string that matches characters in the trim string.
+ * BOTH: removes any characters from both ends of the source string that 
matches characters in the trim string.
+  */
 @ExpressionDescription(
-  usage = "_FUNC_(str) - Removes the leading and trailing space characters 
from `str`.",
+  usage = """
+_FUNC_(str) - Removes the leading and trailing space characters from 
`str`.
+_FUNC_(BOTH trimStr FROM str) - Remove the leading and trailing 
trimString from `str`
+_FUNC_(LEADING trimStr FROM str) - Remove the leading trimString from 
`str`
+_FUNC_(TRAILING trimStr FROM str) - Remove the trailing trimString 
from `str`
+  """,
   extended = """
+Arguments:
+  str - a string expression
+  trimString - the trim string
+  BOTH, FROM - these are keyword to specify for trim string from both 
ends of the string
+  LEADING, FROM - these are keyword to specify for trim string from 
left end of the string
+  TRAILING, FROM - these are keyword to specify for trim string from 
right end of the string
 Examples:
   > SELECT _FUNC_('SparkSQL   ');
SparkSQL
+  > SELECT _FUNC_(BOTH 'SL' FROM 'SSparkSQLS');
+   parkSQ
+  > SELECT _FUNC_(LEADING 'paS' FROM 'SSparkSQLS');
+   rkSQLS
+  > SELECT _FUNC_(TRAILING 'SLQ' FROM 'SSparkSQLS');
+   SSparkS
   """)
-case class StringTrim(child: Expression)
-  extends UnaryExpression with String2StringExpression {
+case class StringTrim(children: Seq[Expression])
+  extends Expression with ImplicitCastInputTypes {
+
+  require(children.size <= 2 && children.nonEmpty,
+s"$prettyName requires at least one argument and no more than two.")
+
+  override def dataType: DataType = StringType
+  override def inputTypes: Seq[AbstractDataType] = 
Seq.fill(children.size)(StringType)
 
-  def convert(v: UTF8String): UTF8String = v.trim()
+  override def nullable: Boolean = children.exists(_.nullable)
+  override def foldable: Boolean = children.forall(_.foldable)
 
   override def prettyName: String = "trim"
 
+  // trim function can take one or two arguments.
+  // For one argument(children size is 1), it is the trim space function.
+  // For two arguments(children size is 2), it is the trim function with 
one of these options: BOTH/LEADING/TRAILING.
+  override def eval(input: InternalRow): Any = {
+val inputs = children.map(_.eval(input).asInstanceOf[UTF8String])
+if (inputs(0) != null) {
+  if (children.size == 1) {
+return inputs(0).trim()
+  } else if (inputs(1) != null) {
+return inputs(1).trim(inputs(0))
+  }
+}
+null
+  }
+
   override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
-defineCodeGen(ctx, ev, c => s"($c).trim()")
+if (children.size == 2 && !children(0).isInstanceOf[Literal]) {
+  throw new AnalysisException(s"The trimming parameter should be 
Literal.")}
+
+val evals = children.map(_.genCode(ctx))
+val inputs = evals.map { eval =>
+  s"${eval.isNull} ? null : ${eval.value}"
+}
+val getTrimFunction = if (children.size == 1) {
+  s"UTF8String ${ev.value} = ${inputs(0)}.trim();"
+} else {
+  s"UTF8String ${ev.value} = 
${inputs(1)}.trim(${inputs(0)});".stripMargin
+}
+ev.copy(evals.map(_.code).mkString("\n") + s"""
+  boolean ${ev.isNull} = false;
+  ${getTrimFunction};
+  if (${ev.value} == null) {
+${ev.isNull} = true;

[GitHub] spark issue #18078: [SPARK-10643] [Core] Make spark-submit download remote f...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18078
  
**[Test build #77435 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77435/testReport)**
 for PR 18078 at commit 
[`2d6f2cd`](https://github.com/apache/spark/commit/2d6f2cdafb30a80ec441ef921238844ace2cd283).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18078: [SPARK-10643] [Core] Make spark-submit download remote f...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18078
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77434/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18078: [SPARK-10643] [Core] Make spark-submit download remote f...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18078
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18078: [SPARK-10643] [Core] Make spark-submit download remote f...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18078
  
**[Test build #77434 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77434/testReport)**
 for PR 18078 at commit 
[`2d6f2cd`](https://github.com/apache/spark/commit/2d6f2cdafb30a80ec441ef921238844ace2cd283).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18126: [SPARK-20843][Core]Add a config to set driver ter...

[BryanCutler]

Github user BryanCutler commented on a diff in the pull request:

https://github.com/apache/spark/pull/18126#discussion_r118803788
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala ---
@@ -57,7 +57,8 @@ private[deploy] class DriverRunner(
   @volatile private[worker] var finalException: Option[Exception] = None
 
   // Timeout to wait for when trying to terminate a driver.
-  private val DRIVER_TERMINATE_TIMEOUT_MS = 10 * 1000
+  private val DRIVER_TERMINATE_TIMEOUT_MS =
+conf.getTimeAsMs("spark.worker.driverTerminateTimeout", "10s")
--- End diff --

Just wondering if maybe adding something to the property to be clear that 
this is for a driver with deploy mode cluster only?  Although it is prefixed 
with `worker` so maybe that is good enough.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18114: [SPARK-20889][SparkR] Grouped documentation for DATETIME...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18114
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18114: [SPARK-20889][SparkR] Grouped documentation for DATETIME...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18114
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77441/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18114: [SPARK-20889][SparkR] Grouped documentation for DATETIME...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18114
  
**[Test build #77441 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77441/testReport)**
 for PR 18114 at commit 
[`944aa92`](https://github.com/apache/spark/commit/944aa92e4c35015010ee9cdaf913f5639670acb6).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `#' @param x if of class Column, it is used to perform arithmatic 
operations with


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18080: [Spark-20771][SQL] Make weekofyear more intuitive

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18080#discussion_r118803850
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
 ---
@@ -402,23 +402,40 @@ case class DayOfMonth(child: Expression) extends 
UnaryExpression with ImplicitCa
   }
 }
 
+// scalastyle:off line.size.limit
 @ExpressionDescription(
-  usage = "_FUNC_(date) - Returns the week of the year of the given date.",
+  usage = "_FUNC_(date[, format]) - Returns the week of the year of the 
given date. Defaults to ISO 8601 standard, but can be gregorian specific",
   extended = """
 Examples:
   > SELECT _FUNC_('2008-02-20');
8
+  > SELECT _FUNC_('2017-01-01', 'gregorian');
+   1
+  > SELECT _FUNC_('2017-01-01', 'iso');
+   52
+  > SELECT _FUNC_('2017-01-01');
+   52
   """)
-case class WeekOfYear(child: Expression) extends UnaryExpression with 
ImplicitCastInputTypes {
+// scalastyle:on line.size.limit
+case class WeekOfYear(child: Expression, format: Expression) extends
+  UnaryExpression with ImplicitCastInputTypes {
+
+  def this(child: Expression) = {
+this(child, Literal("iso"))
+  }
 
   override def inputTypes: Seq[AbstractDataType] = Seq(DateType)
 
   override def dataType: DataType = IntegerType
 
+  @transient private lazy val minimalDays = {
+if ("gregorian".equalsIgnoreCase(format.toString)) 1 else 4
--- End diff --

How many formats the other DB/systems allow? Could you do a search? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org