[GitHub] [spark] shahidki31 commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-17 Thread GitBox
shahidki31 commented on issue #24715: [SPARK-25474][SQL] Data source tables 
support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#issuecomment-522293215
 
 
   If we do 3 times computes statistics like,
   ```
   sql(s"EXPLAIN COST SELECT * FROM spark_25474 '")
   sql(s"EXPLAIN COST SELECT * FROM spark_25474 '")
   sql(s"EXPLAIN COST SELECT * FROM spark_25474 '")
   ```
   Then all the 3 times it falls back to HDFS or only the first time, as the 
table statistics hasn't changed? Then shouldn't we update the table statistics 
after it computes first time, and next time onwards we can directly gets from 
the catalog table?.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] felixcheung commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
felixcheung commented on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522292992
 
 
   can you please check appveyor build clean on this change? 
https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark/builds/26773731 
the last build failed, the next commit you push should trigger it again


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] felixcheung commented on issue #25483: [SPARK-28766][R][DOC] Fix CRAN incoming feasibility warning on invalid URL

2019-08-17 Thread GitBox
felixcheung commented on issue #25483: [SPARK-28766][R][DOC] Fix CRAN incoming 
feasibility warning on invalid URL
URL: https://github.com/apache/spark/pull/25483#issuecomment-522292775
 
 
   LGTM, thanks for fixing


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] 
Convert and port 'pgSQL/join.sql' into UDF test base
URL: https://github.com/apache/spark/pull/25371#issuecomment-522292769
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #25371: 
[SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF 
test base
URL: https://github.com/apache/spark/pull/25371#issuecomment-522292771
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14355/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #25371: 
[SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF 
test base
URL: https://github.com/apache/spark/pull/25371#issuecomment-522292769
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] 
Convert and port 'pgSQL/join.sql' into UDF test base
URL: https://github.com/apache/spark/pull/25371#issuecomment-522292771
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14355/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base

2019-08-17 Thread GitBox
SparkQA commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert 
and port 'pgSQL/join.sql' into UDF test base
URL: https://github.com/apache/spark/pull/25371#issuecomment-522292549
 
 
   **[Test build #109288 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109288/testReport)**
 for PR 25371 at commit 
[`a750727`](https://github.com/apache/spark/commit/a750727bd1a2e785b49676708a6002f97eecac0c).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #22282: [SPARK-23539][SS] Add support 
for Kafka headers in Structured Streaming
URL: https://github.com/apache/spark/pull/22282#issuecomment-522290762
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109287/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #22282: [SPARK-23539][SS] Add support 
for Kafka headers in Structured Streaming
URL: https://github.com/apache/spark/pull/22282#issuecomment-522290759
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming

2019-08-17 Thread GitBox
SparkQA removed a comment on issue #22282: [SPARK-23539][SS] Add support for 
Kafka headers in Structured Streaming
URL: https://github.com/apache/spark/pull/22282#issuecomment-522289801
 
 
   **[Test build #109287 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109287/testReport)**
 for PR 22282 at commit 
[`e413957`](https://github.com/apache/spark/commit/e413957f7364076249362c5de142609cddbcd193).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #22282: [SPARK-23539][SS] Add support for 
Kafka headers in Structured Streaming
URL: https://github.com/apache/spark/pull/22282#issuecomment-522290762
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109287/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming

2019-08-17 Thread GitBox
SparkQA commented on issue #22282: [SPARK-23539][SS] Add support for Kafka 
headers in Structured Streaming
URL: https://github.com/apache/spark/pull/22282#issuecomment-522290743
 
 
   **[Test build #109287 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109287/testReport)**
 for PR 22282 at commit 
[`e413957`](https://github.com/apache/spark/commit/e413957f7364076249362c5de142609cddbcd193).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #22282: [SPARK-23539][SS] Add support for 
Kafka headers in Structured Streaming
URL: https://github.com/apache/spark/pull/22282#issuecomment-522290759
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] viirya commented on a change in pull request #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base

2019-08-17 Thread GitBox
viirya commented on a change in pull request #25371: 
[SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF 
test base
URL: https://github.com/apache/spark/pull/25371#discussion_r314967775
 
 

 ##
 File path: sql/core/src/test/resources/sql-tests/inputs/udf/pgSQL/udf-join.sql
 ##
 @@ -0,0 +1,2081 @@
+--
+-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+--
+--
+-- JOIN
+-- Test JOIN clauses
+-- 
https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/sql/join.sql
+--
+-- This test file was converted from pgSQL/join.sql.
+
+CREATE OR REPLACE TEMPORARY VIEW INT4_TBL AS SELECT * FROM
+  (VALUES (0), (123456), (-123456), (2147483647), (-2147483647))
+  AS v(f1);
+CREATE OR REPLACE TEMPORARY VIEW INT8_TBL AS SELECT * FROM
+  (VALUES
+(123, 456),
+(123, 4567890123456789),
+(4567890123456789, 123),
+(4567890123456789, 4567890123456789),
+(4567890123456789, -4567890123456789))
+  AS v(q1, q2);
+CREATE OR REPLACE TEMPORARY VIEW FLOAT8_TBL AS SELECT * FROM
+  (VALUES (0.0), (1004.30), (-34.84),
+(cast('1.2345678901234e+200' as double)), (cast('1.2345678901234e-200' as 
double)))
+  AS v(f1);
+CREATE OR REPLACE TEMPORARY VIEW TEXT_TBL AS SELECT * FROM
+  (VALUES ('doh!'), ('hi de ho neighbor'))
+  AS v(f1);
+CREATE OR REPLACE TEMPORARY VIEW tenk2 AS SELECT * FROM tenk1;
+
+CREATE TABLE J1_TBL (
+  i integer,
+  j integer,
+  t string
+) USING parquet;
+
+CREATE TABLE J2_TBL (
+  i integer,
+  k integer
+) USING parquet;
+
+
+INSERT INTO J1_TBL VALUES (1, 4, 'one');
+INSERT INTO J1_TBL VALUES (2, 3, 'two');
+INSERT INTO J1_TBL VALUES (3, 2, 'three');
+INSERT INTO J1_TBL VALUES (4, 1, 'four');
+INSERT INTO J1_TBL VALUES (5, 0, 'five');
+INSERT INTO J1_TBL VALUES (6, 6, 'six');
+INSERT INTO J1_TBL VALUES (7, 7, 'seven');
+INSERT INTO J1_TBL VALUES (8, 8, 'eight');
+INSERT INTO J1_TBL VALUES (0, NULL, 'zero');
+INSERT INTO J1_TBL VALUES (NULL, NULL, 'null');
+INSERT INTO J1_TBL VALUES (NULL, 0, 'zero');
+
+INSERT INTO J2_TBL VALUES (1, -1);
+INSERT INTO J2_TBL VALUES (2, 2);
+INSERT INTO J2_TBL VALUES (3, -3);
+INSERT INTO J2_TBL VALUES (2, 4);
+INSERT INTO J2_TBL VALUES (5, -5);
+INSERT INTO J2_TBL VALUES (5, -5);
+INSERT INTO J2_TBL VALUES (0, NULL);
+INSERT INTO J2_TBL VALUES (NULL, NULL);
+INSERT INTO J2_TBL VALUES (NULL, 0);
+
+-- [SPARK-20856] Do not need onerow because it only used for test statement 
using nested joins
+-- useful in some tests below
+-- create temp table onerow();
+-- insert into onerow default values;
+-- analyze onerow;
+
+
+--
+-- CORRELATION NAMES
+-- Make sure that table/column aliases are supported
+-- before diving into more complex join syntax.
+--
+
+SELECT udf('') AS `xxx`, udf(i), udf(j), udf(t)
+  FROM J1_TBL AS tx;
+
+SELECT udf(udf('')) AS `xxx`, udf(udf(i)), udf(j), udf(t)
+  FROM J1_TBL tx;
+
+SELECT udf('') AS `xxx`, a, udf(udf(b)), c
+  FROM J1_TBL AS t1 (a, b, c);
+
+SELECT udf('') AS `xxx`, udf(a), udf(b), udf(udf(c))
+  FROM J1_TBL t1 (a, b, c);
+
+SELECT udf('') AS `xxx`, udf(a), b, udf(c), udf(d), e
+  FROM J1_TBL t1 (a, b, c), J2_TBL t2 (d, e);
+
+-- [SPARK-28377] Fully support correlation names in the FROM clause
+-- SELECT '' AS "xxx", t1.a, t2.e
+--   FROM J1_TBL t1 (a, b, c), J2_TBL t2 (d, e)
+--   WHERE t1.a = t2.d;
+
+
+--
+-- CROSS JOIN
+-- Qualifications are not allowed on cross joins,
+-- which degenerate into a standard unqualified inner join.
+--
+
+SELECT udf('') AS `xxx`, *
+  FROM J1_TBL CROSS JOIN J2_TBL;
+
+-- ambiguous column
+SELECT udf('') AS `xxx`, udf(i), udf(k), udf(t)
+  FROM J1_TBL CROSS JOIN J2_TBL;
+
+-- resolve previous ambiguity by specifying the table name
+SELECT udf('') AS `xxx`, udf(t1.i), udf(k), udf(t)
+  FROM J1_TBL t1 CROSS JOIN J2_TBL t2;
+
+SELECT udf(udf('')) AS `xxx`, udf(udf(ii)), udf(udf(tt)), udf(udf(kk))
+  FROM (J1_TBL CROSS JOIN J2_TBL)
+AS tx (ii, jj, tt, ii2, kk);
+
+-- [SPARK-28377] Fully support correlation names in the FROM clause
+-- SELECT '' AS `xxx`, tx.ii, tx.jj, tx.kk
+--   FROM (J1_TBL t1 (a, b, c) CROSS JOIN J2_TBL t2 (d, e))
+-- AS tx (ii, jj, tt, ii2, kk);
+
+SELECT udf('') AS `xxx`, udf(udf(j1_tbl.i)), udf(j), udf(t), udf(a.i), 
udf(a.k), udf(b.i),  udf(b.k)
+  FROM J1_TBL CROSS JOIN J2_TBL a CROSS JOIN J2_TBL b;
+
+
+--
+--
+-- Inner joins (equi-joins)
+--
+--
+
+--
+-- Inner joins (equi-joins) with USING clause
+-- The USING syntax changes the shape of the resulting table
+-- by including a column in the USING clause only once in the result.
+--
+
+-- Inner equi-join on specified column
+SELECT udf('') AS `xxx`, udf(i), udf(j), udf(t), udf(k)
+  FROM J1_TBL INNER JOIN J2_TBL USING (i);
+
+-- Same as above, slightly different syntax
+SELECT udf(udf('')) AS `xxx`, udf(i), udf(j), udf(t), udf(k)
+  FROM J1_TBL JOIN J2_TBL USING (i);
+
+SELECT udf('') AS `xxx`, *
+  FROM J1_TBL t1 (a, b, c) JOIN J2_TBL t2 (a, d) USING (a)
+  ORDER BY udf(udf(a)), udf(d);
+
+-- [SPARK-28377] Fully support correlation 

[GitHub] [spark] viirya commented on a change in pull request #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base

2019-08-17 Thread GitBox
viirya commented on a change in pull request #25371: 
[SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF 
test base
URL: https://github.com/apache/spark/pull/25371#discussion_r314967707
 
 

 ##
 File path: sql/core/src/test/resources/sql-tests/inputs/udf/pgSQL/udf-join.sql
 ##
 @@ -0,0 +1,2081 @@
+--
+-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+--
+--
+-- JOIN
+-- Test JOIN clauses
+-- 
https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/sql/join.sql
+--
+-- This test file was converted from pgSQL/join.sql.
+
+CREATE OR REPLACE TEMPORARY VIEW INT4_TBL AS SELECT * FROM
+  (VALUES (0), (123456), (-123456), (2147483647), (-2147483647))
+  AS v(f1);
+CREATE OR REPLACE TEMPORARY VIEW INT8_TBL AS SELECT * FROM
+  (VALUES
+(123, 456),
+(123, 4567890123456789),
+(4567890123456789, 123),
+(4567890123456789, 4567890123456789),
+(4567890123456789, -4567890123456789))
+  AS v(q1, q2);
+CREATE OR REPLACE TEMPORARY VIEW FLOAT8_TBL AS SELECT * FROM
+  (VALUES (0.0), (1004.30), (-34.84),
+(cast('1.2345678901234e+200' as double)), (cast('1.2345678901234e-200' as 
double)))
+  AS v(f1);
+CREATE OR REPLACE TEMPORARY VIEW TEXT_TBL AS SELECT * FROM
+  (VALUES ('doh!'), ('hi de ho neighbor'))
+  AS v(f1);
+CREATE OR REPLACE TEMPORARY VIEW tenk2 AS SELECT * FROM tenk1;
+
+CREATE TABLE J1_TBL (
+  i integer,
+  j integer,
+  t string
+) USING parquet;
+
+CREATE TABLE J2_TBL (
+  i integer,
+  k integer
+) USING parquet;
+
+
+INSERT INTO J1_TBL VALUES (1, 4, 'one');
+INSERT INTO J1_TBL VALUES (2, 3, 'two');
+INSERT INTO J1_TBL VALUES (3, 2, 'three');
+INSERT INTO J1_TBL VALUES (4, 1, 'four');
+INSERT INTO J1_TBL VALUES (5, 0, 'five');
+INSERT INTO J1_TBL VALUES (6, 6, 'six');
+INSERT INTO J1_TBL VALUES (7, 7, 'seven');
+INSERT INTO J1_TBL VALUES (8, 8, 'eight');
+INSERT INTO J1_TBL VALUES (0, NULL, 'zero');
+INSERT INTO J1_TBL VALUES (NULL, NULL, 'null');
+INSERT INTO J1_TBL VALUES (NULL, 0, 'zero');
+
+INSERT INTO J2_TBL VALUES (1, -1);
+INSERT INTO J2_TBL VALUES (2, 2);
+INSERT INTO J2_TBL VALUES (3, -3);
+INSERT INTO J2_TBL VALUES (2, 4);
+INSERT INTO J2_TBL VALUES (5, -5);
+INSERT INTO J2_TBL VALUES (5, -5);
+INSERT INTO J2_TBL VALUES (0, NULL);
+INSERT INTO J2_TBL VALUES (NULL, NULL);
+INSERT INTO J2_TBL VALUES (NULL, 0);
+
+-- [SPARK-20856] Do not need onerow because it only used for test statement 
using nested joins
+-- useful in some tests below
+-- create temp table onerow();
+-- insert into onerow default values;
+-- analyze onerow;
+
+
+--
+-- CORRELATION NAMES
+-- Make sure that table/column aliases are supported
+-- before diving into more complex join syntax.
+--
+
+SELECT udf('') AS `xxx`, udf(i), udf(j), udf(t)
+  FROM J1_TBL AS tx;
+
+SELECT udf(udf('')) AS `xxx`, udf(udf(i)), udf(j), udf(t)
+  FROM J1_TBL tx;
+
+SELECT udf('') AS `xxx`, a, udf(udf(b)), c
+  FROM J1_TBL AS t1 (a, b, c);
+
+SELECT udf('') AS `xxx`, udf(a), udf(b), udf(udf(c))
+  FROM J1_TBL t1 (a, b, c);
+
+SELECT udf('') AS `xxx`, udf(a), b, udf(c), udf(d), e
+  FROM J1_TBL t1 (a, b, c), J2_TBL t2 (d, e);
+
+-- [SPARK-28377] Fully support correlation names in the FROM clause
+-- SELECT '' AS "xxx", t1.a, t2.e
+--   FROM J1_TBL t1 (a, b, c), J2_TBL t2 (d, e)
+--   WHERE t1.a = t2.d;
+
+
+--
+-- CROSS JOIN
+-- Qualifications are not allowed on cross joins,
+-- which degenerate into a standard unqualified inner join.
+--
+
+SELECT udf('') AS `xxx`, *
+  FROM J1_TBL CROSS JOIN J2_TBL;
+
+-- ambiguous column
+SELECT udf('') AS `xxx`, udf(i), udf(k), udf(t)
+  FROM J1_TBL CROSS JOIN J2_TBL;
+
+-- resolve previous ambiguity by specifying the table name
+SELECT udf('') AS `xxx`, udf(t1.i), udf(k), udf(t)
+  FROM J1_TBL t1 CROSS JOIN J2_TBL t2;
+
+SELECT udf(udf('')) AS `xxx`, udf(udf(ii)), udf(udf(tt)), udf(udf(kk))
+  FROM (J1_TBL CROSS JOIN J2_TBL)
+AS tx (ii, jj, tt, ii2, kk);
+
+-- [SPARK-28377] Fully support correlation names in the FROM clause
+-- SELECT '' AS `xxx`, tx.ii, tx.jj, tx.kk
+--   FROM (J1_TBL t1 (a, b, c) CROSS JOIN J2_TBL t2 (d, e))
+-- AS tx (ii, jj, tt, ii2, kk);
+
+SELECT udf('') AS `xxx`, udf(udf(j1_tbl.i)), udf(j), udf(t), udf(a.i), 
udf(a.k), udf(b.i),  udf(b.k)
+  FROM J1_TBL CROSS JOIN J2_TBL a CROSS JOIN J2_TBL b;
+
+
+--
+--
+-- Inner joins (equi-joins)
+--
+--
+
+--
+-- Inner joins (equi-joins) with USING clause
+-- The USING syntax changes the shape of the resulting table
+-- by including a column in the USING clause only once in the result.
+--
+
+-- Inner equi-join on specified column
+SELECT udf('') AS `xxx`, udf(i), udf(j), udf(t), udf(k)
+  FROM J1_TBL INNER JOIN J2_TBL USING (i);
+
+-- Same as above, slightly different syntax
+SELECT udf(udf('')) AS `xxx`, udf(i), udf(j), udf(t), udf(k)
+  FROM J1_TBL JOIN J2_TBL USING (i);
+
+SELECT udf('') AS `xxx`, *
+  FROM J1_TBL t1 (a, b, c) JOIN J2_TBL t2 (a, d) USING (a)
+  ORDER BY udf(udf(a)), udf(d);
+
+-- [SPARK-28377] Fully support correlation 

[GitHub] [spark] AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522289868
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109286/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522289867
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
SparkQA commented on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522289863
 
 
   **[Test build #109286 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109286/testReport)**
 for PR 22570 at commit 
[`c5dc394`](https://github.com/apache/spark/commit/c5dc394acc208051084d9257a1da6a82adb9215c).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
SparkQA removed a comment on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522289805
 
 
   **[Test build #109286 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109286/testReport)**
 for PR 22570 at commit 
[`c5dc394`](https://github.com/apache/spark/commit/c5dc394acc208051084d9257a1da6a82adb9215c).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522289867
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522289868
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109286/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming

2019-08-17 Thread GitBox
SparkQA commented on issue #22282: [SPARK-23539][SS] Add support for Kafka 
headers in Structured Streaming
URL: https://github.com/apache/spark/pull/22282#issuecomment-522289801
 
 
   **[Test build #109287 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109287/testReport)**
 for PR 22282 at commit 
[`e413957`](https://github.com/apache/spark/commit/e413957f7364076249362c5de142609cddbcd193).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
SparkQA commented on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522289805
 
 
   **[Test build #109286 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109286/testReport)**
 for PR 22570 at commit 
[`c5dc394`](https://github.com/apache/spark/commit/c5dc394acc208051084d9257a1da6a82adb9215c).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522289720
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14354/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522289719
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjinleekr commented on a change in pull request #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming

2019-08-17 Thread GitBox
dongjinleekr commented on a change in pull request #22282: [SPARK-23539][SS] 
Add support for Kafka headers in Structured Streaming
URL: https://github.com/apache/spark/pull/22282#discussion_r314967556
 
 

 ##
 File path: 
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDataConsumerSuite.scala
 ##
 @@ -105,7 +105,17 @@ class KafkaDataConsumerSuite extends SharedSQLContext 
with PrivateMethodTester {
   val headers = record.headers().toArray.map(header => (header.key(), 
header.value())).toSeq
   (value, headers)
 }
-data === rcvd
+data zip rcvd foreach { case (expected, actual) =>
 
 Review comment:
   @srowen Got it. Here is the fix.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522289720
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14354/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522289719
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found

2019-08-17 Thread GitBox
SparkQA commented on issue #25399: [SPARK-28670][SQL] create function should 
thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#issuecomment-522287253
 
 
   **[Test build #109285 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109285/testReport)**
 for PR 25399 at commit 
[`e7f5d46`](https://github.com/apache/spark/commit/e7f5d461600b6f5250b60910b2b0061090774800).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #25399: [SPARK-28670][SQL] create 
function should thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#issuecomment-522287161
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #25399: [SPARK-28670][SQL] create 
function should thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#issuecomment-522287163
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14353/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #25399: [SPARK-28670][SQL] create function 
should thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#issuecomment-522287161
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #25399: [SPARK-28670][SQL] create function 
should thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#issuecomment-522287163
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14353/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] gatorsmile commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base

2019-08-17 Thread GitBox
gatorsmile commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert 
and port 'pgSQL/join.sql' into UDF test base
URL: https://github.com/apache/spark/pull/25371#issuecomment-522286594
 
 
   cc @HyukjinKwon 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] sandeep-katta commented on issue #25452: [SPARK-28710][SQL]to fix replace function, spark should call drop and create function

2019-08-17 Thread GitBox
sandeep-katta commented on issue #25452: [SPARK-28710][SQL]to fix replace 
function, spark should call drop and create function
URL: https://github.com/apache/spark/pull/25452#issuecomment-522285894
 
 
   > I left a few comments, @sandeep-katta . Thank you for your contribution.
   
   I have fixed the comments


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] sandeep-katta commented on a change in pull request #25427: [SPARK-28705][SQL][TEST] Drop tables after being used in AnalysisExternalCatalogSuite

2019-08-17 Thread GitBox
sandeep-katta commented on a change in pull request #25427: 
[SPARK-28705][SQL][TEST] Drop tables after being used in 
AnalysisExternalCatalogSuite
URL: https://github.com/apache/spark/pull/25427#discussion_r314966127
 
 

 ##
 File path: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisExternalCatalogSuite.scala
 ##
 @@ -28,13 +29,28 @@ import org.apache.spark.sql.catalyst.expressions.{Alias, 
AttributeReference}
 import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, Project}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.types._
+import org.apache.spark.util.Utils
 
 class AnalysisExternalCatalogSuite extends AnalysisTest with Matchers {
+  var tempDir: File = _
+
+  override def beforeAll(): Unit = {
+super.beforeAll()
+tempDir = Utils.createTempDir()
+  }
+
+  override def afterEach: Unit = {
+try {
+  Utils.deleteRecursively(tempDir)
+} finally {
+  super.afterEach()
+}
+  }
 
 Review comment:
   clean up should happen after each test case execution, so only I ve put in 
the forEach


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow

2019-08-17 Thread GitBox
maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw 
exceptions when casting to integers causes overflow
URL: https://github.com/apache/spark/pull/25461#discussion_r314966093
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ##
 @@ -1279,8 +1409,26 @@ case class Cast(child: Expression, dataType: DataType, 
timeZoneId: Option[String
   (c, evPrim, evNull) => code"$evNull = true;"
 case TimestampType =>
   (c, evPrim, evNull) => code"$evPrim = (long) 
${timestampToIntegerCode(c)};"
+case DecimalType() if failOnIntegerOverflow =>
+  (c, evPrim, evNull) =>
+code"""
+  try {
+$evPrim = $c.toJavaBigInteger().longValueExact();
+  } catch (ArithmeticException e) {
+throw new ArithmeticException("Casting $c to Long causes 
overflow");
 
 Review comment:
   It seems `longValueExact` throws an exception even when the non-overflow 
case detected: 
https://github.com/bpupadhyaya/openjdk-8/blob/master/jdk/src/share/classes/java/math/BigDecimal.java#L3088


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] gatorsmile closed pull request #25373: [SPARK-28527][SQL][TEST] Re-run all the tests in SQLQueryTestSuite via Thrift Server

2019-08-17 Thread GitBox
gatorsmile closed pull request #25373: [SPARK-28527][SQL][TEST] Re-run all the 
tests in SQLQueryTestSuite via Thrift Server
URL: https://github.com/apache/spark/pull/25373
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] gatorsmile commented on issue #25373: [SPARK-28527][SQL][TEST] Re-run all the tests in SQLQueryTestSuite via Thrift Server

2019-08-17 Thread GitBox
gatorsmile commented on issue #25373: [SPARK-28527][SQL][TEST] Re-run all the 
tests in SQLQueryTestSuite via Thrift Server
URL: https://github.com/apache/spark/pull/25373#issuecomment-522284753
 
 
   Thanks! Merged to master.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow

2019-08-17 Thread GitBox
maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw 
exceptions when casting to integers causes overflow
URL: https://github.com/apache/spark/pull/25461#discussion_r314964935
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ##
 @@ -475,7 +478,16 @@ case class Cast(child: Expression, dataType: DataType, 
timeZoneId: Option[String
 case DateType =>
   buildCast[Int](_, d => null)
 case TimestampType =>
-  buildCast[Long](_, t => timestampToLong(t).toInt)
+  buildCast[Long](_, t => {
 
 Review comment:
   How about this? (I think we can fix the other patterns too in the same way
   ```
 if (failOnIntegerOverflow) {
   buildCast[Long](_, t => timestampToLong(t).toInt)
 } else {
   buildCast[Long](_, t => LongExactNumeric.toInt(timestampToLong(t)))
 }
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow

2019-08-17 Thread GitBox
maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw 
exceptions when casting to integers causes overflow
URL: https://github.com/apache/spark/pull/25461#discussion_r314964935
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ##
 @@ -475,7 +478,16 @@ case class Cast(child: Expression, dataType: DataType, 
timeZoneId: Option[String
 case DateType =>
   buildCast[Int](_, d => null)
 case TimestampType =>
-  buildCast[Long](_, t => timestampToLong(t).toInt)
+  buildCast[Long](_, t => {
 
 Review comment:
   How about this? (I think we can fix the other patterns in the same way
   ```
 if (failOnIntegerOverflow) {
   buildCast[Long](_, t => timestampToLong(t).toInt)
 } else {
   buildCast[Long](_, t => LongExactNumeric.toInt(timestampToLong(t)))
 }
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow

2019-08-17 Thread GitBox
maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw 
exceptions when casting to integers causes overflow
URL: https://github.com/apache/spark/pull/25461#discussion_r314964814
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ##
 @@ -461,6 +462,8 @@ case class Cast(child: Expression, dataType: DataType, 
timeZoneId: Option[String
   buildCast[Int](_, d => null)
 case TimestampType =>
   buildCast[Long](_, t => timestampToLong(t))
+case x: NumericType if failOnIntegerOverflow =>
+  b => x.exactNumeric.asInstanceOf[Numeric[Any]].toLong(b)
 
 Review comment:
   How about this form below?
   ```
   case x: NumericType =>
 if (failOnIntegerOverflow) {
   b => x.exactNumeric.asInstanceOf[Numeric[Any]].toLong(b)
 } else {
   b => x.numeric.asInstanceOf[Numeric[Any]].toLong(b)
 }
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow

2019-08-17 Thread GitBox
maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw 
exceptions when casting to integers causes overflow
URL: https://github.com/apache/spark/pull/25461#discussion_r314965182
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ##
 @@ -494,7 +506,27 @@ case class Cast(child: Expression, dataType: DataType, 
timeZoneId: Option[String
 case DateType =>
   buildCast[Int](_, d => null)
 case TimestampType =>
-  buildCast[Long](_, t => timestampToLong(t).toShort)
+  buildCast[Long](_, t => {
+val longValue = timestampToLong(t)
+if (!failOnIntegerOverflow || longValue == longValue.toShort) {
+  longValue.toShort
+} else {
+  throw new ArithmeticException(s"Casting $t to Short causes 
overflow.")
+}
+  })
+case x: NumericType if failOnIntegerOverflow =>
+  b =>
+val intValue = try {
+  x.exactNumeric.asInstanceOf[Numeric[Any]].toInt(b)
 
 Review comment:
   Why do you cast it into int once?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source

2019-08-17 Thread GitBox
SparkQA commented on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS 
fail with Unable to move source
URL: https://github.com/apache/spark/pull/25364#issuecomment-522282658
 
 
   **[Test build #4836 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4836/testReport)**
 for PR 25364 at commit 
[`52421d1`](https://github.com/apache/spark/commit/52421d1850e3205d60d59e150b722f72f8a1feba).
* This patch passes all tests.
* This patch **does not merge cleanly**.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source

2019-08-17 Thread GitBox
SparkQA removed a comment on issue #25364: [SPARK-21067][DOC] Fix Thrift Server 
- CTAS fail with Unable to move source
URL: https://github.com/apache/spark/pull/25364#issuecomment-522276526
 
 
   **[Test build #4836 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4836/testReport)**
 for PR 25364 at commit 
[`52421d1`](https://github.com/apache/spark/commit/52421d1850e3205d60d59e150b722f72f8a1feba).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25485: [SPARK-28434][TESTS][ML] Fix values in dummy tree in DecisionTreeSuite

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #25485: [SPARK-28434][TESTS][ML] Fix values in 
dummy tree in DecisionTreeSuite
URL: https://github.com/apache/spark/pull/25485#issuecomment-522282473
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25485: [SPARK-28434][TESTS][ML] Fix values in dummy tree in DecisionTreeSuite

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #25485: [SPARK-28434][TESTS][ML] Fix values in 
dummy tree in DecisionTreeSuite
URL: https://github.com/apache/spark/pull/25485#issuecomment-522282474
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109283/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25485: [SPARK-28434][TESTS][ML] Fix values in dummy tree in DecisionTreeSuite

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #25485: [SPARK-28434][TESTS][ML] Fix 
values in dummy tree in DecisionTreeSuite
URL: https://github.com/apache/spark/pull/25485#issuecomment-522282474
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109283/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25485: [SPARK-28434][TESTS][ML] Fix values in dummy tree in DecisionTreeSuite

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #25485: [SPARK-28434][TESTS][ML] Fix 
values in dummy tree in DecisionTreeSuite
URL: https://github.com/apache/spark/pull/25485#issuecomment-522282473
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25485: [SPARK-28434][TESTS][ML] Fix values in dummy tree in DecisionTreeSuite

2019-08-17 Thread GitBox
SparkQA commented on issue #25485: [SPARK-28434][TESTS][ML] Fix values in dummy 
tree in DecisionTreeSuite
URL: https://github.com/apache/spark/pull/25485#issuecomment-522282440
 
 
   **[Test build #109283 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109283/testReport)**
 for PR 25485 at commit 
[`a8472e2`](https://github.com/apache/spark/commit/a8472e2774deac565825406828d35834af40cba5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #25485: [SPARK-28434][TESTS][ML] Fix values in dummy tree in DecisionTreeSuite

2019-08-17 Thread GitBox
SparkQA removed a comment on issue #25485: [SPARK-28434][TESTS][ML] Fix values 
in dummy tree in DecisionTreeSuite
URL: https://github.com/apache/spark/pull/25485#issuecomment-522279633
 
 
   **[Test build #109283 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109283/testReport)**
 for PR 25485 at commit 
[`a8472e2`](https://github.com/apache/spark/commit/a8472e2774deac565825406828d35834af40cba5).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522282101
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109282/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522282100
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
SparkQA removed a comment on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522277856
 
 
   **[Test build #109282 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109282/testReport)**
 for PR 22570 at commit 
[`24f95b7`](https://github.com/apache/spark/commit/24f95b72930fe2426c3f639115d7be88dec87925).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522282101
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109282/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522282100
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
SparkQA commented on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522282043
 
 
   **[Test build #109282 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109282/testReport)**
 for PR 22570 at commit 
[`24f95b7`](https://github.com/apache/spark/commit/24f95b72930fe2426c3f639115d7be88dec87925).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found

2019-08-17 Thread GitBox
maropu commented on issue #25399: [SPARK-28670][SQL] create function should 
thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#issuecomment-522281974
 
 
   cc: @dongjoon-hyun 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found

2019-08-17 Thread GitBox
maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create 
function should thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#discussion_r314964670
 
 

 ##
 File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
 ##
 @@ -3180,6 +3180,20 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
 }
 
   }
+
+  test("SPARK-28670: create function should throw AnalysisException if UDF 
classes not found") {
+Seq(true, false).foreach(isTemporary => {
 
 Review comment:
   nit: `Seq(true, false).foreach { isTemporary =>`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found

2019-08-17 Thread GitBox
maropu commented on issue #25399: [SPARK-28670][SQL] create function should 
thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#issuecomment-522281927
 
 
   yea, please update it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-17 Thread GitBox
SparkQA commented on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-522281589
 
 
   **[Test build #109284 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109284/testReport)**
 for PR 25443 at commit 
[`3856828`](https://github.com/apache/spark/commit/385682881f097fe80d88ec2608541ad67ee6a11a).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-522281527
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14352/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-522281526
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-522281526
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-522281527
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14352/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #25033: [SPARK-28224][SQL] Check overflow in decimal Sum aggregate

2019-08-17 Thread GitBox
maropu commented on a change in pull request #25033: [SPARK-28224][SQL] Check 
overflow in decimal Sum aggregate
URL: https://github.com/apache/spark/pull/25033#discussion_r314962435
 
 

 ##
 File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
 ##
 @@ -156,6 +156,33 @@ class DataFrameSuite extends QueryTest with 
SharedSQLContext {
   structDf.select(xxhash64($"a", $"record.*")))
   }
 
+  test("SPARK-28224: Aggregate sum big decimal overflow") {
+Seq(true, false).foreach { nullOnOverflow =>
+  withSQLConf((SQLConf.DECIMAL_OPERATIONS_NULL_ON_OVERFLOW.key, 
nullOnOverflow.toString)) {
+val structDf = largeDecimals.select("a").agg(sum("a"))
+if (nullOnOverflow) {
+  checkAnswer(structDf, Row(null))
+} else {
+  val e = intercept[Exception] {
+structDf.collect
+  }
+  assert(e.getCause.getClass.equals(classOf[ArithmeticException]))
+  assert(e.getCause.getMessage.contains("cannot be represented as 
Decimal"))
+}
+  }
+}
+  }
+
+  test("Aggregate sum integers") {
 
 Review comment:
   But, the case for int/double is an existing path in master, so we already 
have tests for that somewhere, right? I personally think the test above is 
enough for this pr. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-17 Thread GitBox
maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314964437
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
 ##
 @@ -650,4 +650,46 @@ class StatisticsCollectionSuite extends 
StatisticsCollectionTestBase with Shared
   }
 }
   }
+
+  test("Non-partitioned data source table support fallback to HDFS for size 
estimation") {
+withTempDir { dir =>
+  Seq(false, true).foreach { fallBackToHDFS =>
+withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> 
s"$fallBackToHDFS") {
+  withTable("spark_25474") {
+sql(s"CREATE TABLE spark_25474 (c1 BIGINT) USING PARQUET LOCATION 
'${dir.toURI}'")
+
spark.range(5).write.mode(SaveMode.Overwrite).parquet(dir.getCanonicalPath)
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation = 
spark.table("spark_25474").queryExecution.analyzed.children.head
+// fallBackToHDFS = true: The table stats will be recalculated by 
DetermineTableStats
+// fallBackToHDFS = false: The table stats will be recalculated by 
FileIndex
+assert(relation.stats.sizeInBytes === getDataSize(dir))
+  }
+}
+  }
+}
+  }
+
+  test("Partitioned data source table support fallback to HDFS for size 
estimation") {
+Seq(false, true).foreach { fallBackToHDFS =>
+  withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> 
s"$fallBackToHDFS") {
+withTempDir { dir =>
+  withTable("spark_25474") {
+sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
+s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
+sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation = 
spark.table("spark_25474").queryExecution.analyzed.children.head
+if (fallBackToHDFS) {
+  assert(relation.stats.sizeInBytes > 0)
+  assert(relation.stats.sizeInBytes < conf.defaultSizeInBytes)
 
 Review comment:
   We cannot use `getDataSize` here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-17 Thread GitBox
maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314964437
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
 ##
 @@ -650,4 +650,46 @@ class StatisticsCollectionSuite extends 
StatisticsCollectionTestBase with Shared
   }
 }
   }
+
+  test("Non-partitioned data source table support fallback to HDFS for size 
estimation") {
+withTempDir { dir =>
+  Seq(false, true).foreach { fallBackToHDFS =>
+withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> 
s"$fallBackToHDFS") {
+  withTable("spark_25474") {
+sql(s"CREATE TABLE spark_25474 (c1 BIGINT) USING PARQUET LOCATION 
'${dir.toURI}'")
+
spark.range(5).write.mode(SaveMode.Overwrite).parquet(dir.getCanonicalPath)
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation = 
spark.table("spark_25474").queryExecution.analyzed.children.head
+// fallBackToHDFS = true: The table stats will be recalculated by 
DetermineTableStats
+// fallBackToHDFS = false: The table stats will be recalculated by 
FileIndex
+assert(relation.stats.sizeInBytes === getDataSize(dir))
+  }
+}
+  }
+}
+  }
+
+  test("Partitioned data source table support fallback to HDFS for size 
estimation") {
+Seq(false, true).foreach { fallBackToHDFS =>
+  withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> 
s"$fallBackToHDFS") {
+withTempDir { dir =>
+  withTable("spark_25474") {
+sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
+s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
+sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation = 
spark.table("spark_25474").queryExecution.analyzed.children.head
+if (fallBackToHDFS) {
+  assert(relation.stats.sizeInBytes > 0)
+  assert(relation.stats.sizeInBytes < conf.defaultSizeInBytes)
 
 Review comment:
   We cannnot use `getDataSize` here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-17 Thread GitBox
maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314964256
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
 ##
 @@ -619,3 +619,34 @@ object DataSourceStrategy {
 (nonconvertiblePredicates ++ unhandledPredicates, pushedFilters, 
handledFilters)
   }
 }
+
+
+/**
+ * Defines default table statistics if table statistics are not available.
+ */
+class DetermineTableStats(session: SparkSession) extends Rule[LogicalPlan] {
 
 Review comment:
   case class?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-17 Thread GitBox
maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314964375
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
 ##
 @@ -650,4 +650,46 @@ class StatisticsCollectionSuite extends 
StatisticsCollectionTestBase with Shared
   }
 }
   }
+
+  test("Non-partitioned data source table support fallback to HDFS for size 
estimation") {
+withTempDir { dir =>
+  Seq(false, true).foreach { fallBackToHDFS =>
+withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> 
s"$fallBackToHDFS") {
+  withTable("spark_25474") {
+sql(s"CREATE TABLE spark_25474 (c1 BIGINT) USING PARQUET LOCATION 
'${dir.toURI}'")
+
spark.range(5).write.mode(SaveMode.Overwrite).parquet(dir.getCanonicalPath)
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation = 
spark.table("spark_25474").queryExecution.analyzed.children.head
+// fallBackToHDFS = true: The table stats will be recalculated by 
DetermineTableStats
+// fallBackToHDFS = false: The table stats will be recalculated by 
FileIndex
+assert(relation.stats.sizeInBytes === getDataSize(dir))
+  }
+}
+  }
+}
+  }
+
+  test("Partitioned data source table support fallback to HDFS for size 
estimation") {
+Seq(false, true).foreach { fallBackToHDFS =>
+  withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> 
s"$fallBackToHDFS") {
 
 Review comment:
   Just in case, how about explicitly setting `defaultSizeInBytes` here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-17 Thread GitBox
maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314964366
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
 ##
 @@ -650,4 +650,46 @@ class StatisticsCollectionSuite extends 
StatisticsCollectionTestBase with Shared
   }
 }
   }
+
+  test("Non-partitioned data source table support fallback to HDFS for size 
estimation") {
+withTempDir { dir =>
+  Seq(false, true).foreach { fallBackToHDFS =>
+withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> 
s"$fallBackToHDFS") {
+  withTable("spark_25474") {
+sql(s"CREATE TABLE spark_25474 (c1 BIGINT) USING PARQUET LOCATION 
'${dir.toURI}'")
+
spark.range(5).write.mode(SaveMode.Overwrite).parquet(dir.getCanonicalPath)
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation = 
spark.table("spark_25474").queryExecution.analyzed.children.head
+// fallBackToHDFS = true: The table stats will be recalculated by 
DetermineTableStats
+// fallBackToHDFS = false: The table stats will be recalculated by 
FileIndex
 
 Review comment:
   If so, the behaviour of the non-partitioned case should follow that of the 
partitioned case? (If `fallBackToHDFS`=false, we should use 
`defaultSizeInBytes` here, too.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25485: [SPARK-28434][TESTS][ML] Fix values in dummy tree in DecisionTreeSuite

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #25485: [SPARK-28434][TESTS][ML] Fix 
values in dummy tree in DecisionTreeSuite
URL: https://github.com/apache/spark/pull/25485#issuecomment-522279559
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25485: [SPARK-28434][TESTS][ML] Fix values in dummy tree in DecisionTreeSuite

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #25485: [SPARK-28434][TESTS][ML] Fix 
values in dummy tree in DecisionTreeSuite
URL: https://github.com/apache/spark/pull/25485#issuecomment-522279560
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14351/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25485: [SPARK-28434][TESTS][ML] Fix values in dummy tree in DecisionTreeSuite

2019-08-17 Thread GitBox
SparkQA commented on issue #25485: [SPARK-28434][TESTS][ML] Fix values in dummy 
tree in DecisionTreeSuite
URL: https://github.com/apache/spark/pull/25485#issuecomment-522279633
 
 
   **[Test build #109283 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109283/testReport)**
 for PR 25485 at commit 
[`a8472e2`](https://github.com/apache/spark/commit/a8472e2774deac565825406828d35834af40cba5).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25485: [SPARK-28434][TESTS][ML] Fix values in dummy tree in DecisionTreeSuite

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #25485: [SPARK-28434][TESTS][ML] Fix values in 
dummy tree in DecisionTreeSuite
URL: https://github.com/apache/spark/pull/25485#issuecomment-522279559
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25485: [SPARK-28434][TESTS][ML] Fix values in dummy tree in DecisionTreeSuite

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #25485: [SPARK-28434][TESTS][ML] Fix values in 
dummy tree in DecisionTreeSuite
URL: https://github.com/apache/spark/pull/25485#issuecomment-522279560
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14351/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] srowen commented on issue #25187: [SPARK-28434][MLlib] Decision Tree model isn't equal after save and load

2019-08-17 Thread GitBox
srowen commented on issue #25187: [SPARK-28434][MLlib] Decision Tree model 
isn't equal after save and load
URL: https://github.com/apache/spark/pull/25187#issuecomment-522279421
 
 
   I'm closing in favor of https://github.com/apache/spark/pull/25485 which has 
a fix for the tests.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] srowen commented on a change in pull request #25485: [SPARK-28434][TESTS][ML] Fix values in dummy tree in DecisionTreeSuite

2019-08-17 Thread GitBox
srowen commented on a change in pull request #25485: [SPARK-28434][TESTS][ML] 
Fix values in dummy tree in DecisionTreeSuite
URL: https://github.com/apache/spark/pull/25485#discussion_r314963618
 
 

 ##
 File path: 
mllib/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala
 ##
 @@ -561,34 +561,29 @@ object DecisionTreeSuite extends SparkFunSuite {
* Create an internal node with the given node ID and feature type.
* Note: This does NOT set the child nodes.
*/
-  private def createInternalNode(id: Int, featureType: FeatureType): Node = {
+  private def createInternalNode(
+  id: Int, featureType: FeatureType, left: Node, right: Node): Node = {
 val node = Node(nodeIndex = id, new Predict(0.0, 1.0), impurity = 0.5, 
isLeaf = false)
-featureType match {
-  case Continuous =>
-node.split = Some(new Split(feature = 0, threshold = 0.5, Continuous,
-  categories = List.empty[Double]))
-  case Categorical =>
-node.split = Some(new Split(feature = 1, threshold = 0.0, Categorical,
-  categories = List(0.0, 1.0)))
-}
-// TODO: The information gain stats should be consistent with info in 
children: SPARK-7131
-node.stats = Some(new InformationGainStats(gain = 0.1, impurity = 0.2,
-  leftImpurity = 0.3, rightImpurity = 0.4, new Predict(1.0, 0.4), new 
Predict(0.0, 0.6)))
+node.split = Some(featureType match {
+  case Continuous => Split(feature = 0, threshold = 0.5, featureType, 
List.empty[Double])
+  case Categorical => Split(feature = 1, threshold = 0.0, featureType, 
List(0.0, 1.0))
+})
+node.stats = Some(new InformationGainStats(gain = 0.1, impurity = 
node.impurity,
 
 Review comment:
   The changes are really here; above is mostly code cleanup


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] srowen closed pull request #25187: [SPARK-28434][MLlib] Decision Tree model isn't equal after save and load

2019-08-17 Thread GitBox
srowen closed pull request #25187: [SPARK-28434][MLlib] Decision Tree model 
isn't equal after save and load
URL: https://github.com/apache/spark/pull/25187
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] srowen commented on a change in pull request #25485: [SPARK-28434][TESTS][ML] Fix values in dummy tree in DecisionTreeSuite

2019-08-17 Thread GitBox
srowen commented on a change in pull request #25485: [SPARK-28434][TESTS][ML] 
Fix values in dummy tree in DecisionTreeSuite
URL: https://github.com/apache/spark/pull/25485#discussion_r314963624
 
 

 ##
 File path: 
mllib/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala
 ##
 @@ -625,8 +620,7 @@ object DecisionTreeSuite extends SparkFunSuite {
 assert(a.isLeaf === b.isLeaf)
 assert(a.split === b.split)
 (a.stats, b.stats) match {
-  // TODO: Check other fields besides the information gain.
-  case (Some(aStats), Some(bStats)) => assert(aStats.gain === bStats.gain)
+  case (Some(aStats), Some(bStats)) => assert(aStats === bStats)
 
 Review comment:
   This is the improved test


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] srowen opened a new pull request #25485: [SPARK-28434][TESTS][ML] Fix values in dummy tree in DecisionTreeSuite

2019-08-17 Thread GitBox
srowen opened a new pull request #25485: [SPARK-28434][TESTS][ML] Fix values in 
dummy tree in DecisionTreeSuite
URL: https://github.com/apache/spark/pull/25485
 
 
   ### What changes were proposed in this pull request?
   
   Fix dummy tree created in decision tree tests to have actually consistent 
stats, so that it can be compared in tests more completely. The current one has 
values for, say, impurity that don't even match internally.
   
   With this, the tests can assert more about stats staying correct after load.
   
   ### Why are the changes needed?
   
   Fixes a TODO and improves the test slightly.
   
   ### Does this PR introduce any user-facing change?
   
   None
   
   ### How was this patch tested?
   
   Existing tests.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25484: [SPARK-28647][WEBUI] Recover additional metric feature for branch-2.4

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #25484: [SPARK-28647][WEBUI] Recover 
additional metric feature for branch-2.4
URL: https://github.com/apache/spark/pull/25484#issuecomment-522278407
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25484: [SPARK-28647][WEBUI] Recover additional metric feature for branch-2.4

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #25484: [SPARK-28647][WEBUI] Recover 
additional metric feature for branch-2.4
URL: https://github.com/apache/spark/pull/25484#issuecomment-522278410
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109281/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25484: [SPARK-28647][WEBUI] Recover additional metric feature for branch-2.4

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #25484: [SPARK-28647][WEBUI] Recover 
additional metric feature for branch-2.4
URL: https://github.com/apache/spark/pull/25484#issuecomment-522278410
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109281/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25484: [SPARK-28647][WEBUI] Recover additional metric feature for branch-2.4

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #25484: [SPARK-28647][WEBUI] Recover 
additional metric feature for branch-2.4
URL: https://github.com/apache/spark/pull/25484#issuecomment-522278407
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #25484: [SPARK-28647][WEBUI] Recover additional metric feature for branch-2.4

2019-08-17 Thread GitBox
SparkQA removed a comment on issue #25484: [SPARK-28647][WEBUI] Recover 
additional metric feature for branch-2.4
URL: https://github.com/apache/spark/pull/25484#issuecomment-522262551
 
 
   **[Test build #109281 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109281/testReport)**
 for PR 25484 at commit 
[`b7d381d`](https://github.com/apache/spark/commit/b7d381da01ef4327f35bc6c527f99d1e288b).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25484: [SPARK-28647][WEBUI] Recover additional metric feature for branch-2.4

2019-08-17 Thread GitBox
SparkQA commented on issue #25484: [SPARK-28647][WEBUI] Recover additional 
metric feature for branch-2.4
URL: https://github.com/apache/spark/pull/25484#issuecomment-522278357
 
 
   **[Test build #109281 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109281/testReport)**
 for PR 25484 at commit 
[`b7d381d`](https://github.com/apache/spark/commit/b7d381da01ef4327f35bc6c527f99d1e288b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522278077
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522278078
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14350/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522278077
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522278078
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14350/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates

2019-08-17 Thread GitBox
maropu commented on a change in pull request #25410: [SPARK-28690][SQL] Add 
`date_part` function for timestamps/dates
URL: https://github.com/apache/spark/pull/25410#discussion_r314962673
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
 ##
 @@ -1409,48 +1409,7 @@ class AstBuilder(conf: SQLConf) extends 
SqlBaseBaseVisitor[AnyRef] with Logging
* Create a Extract expression.
*/
   override def visitExtract(ctx: ExtractContext): Expression = withOrigin(ctx) 
{
-ctx.field.getText.toUpperCase(Locale.ROOT) match {
-  case "MILLENNIUM" | "MILLENNIA" | "MIL" | "MILS" =>
-Millennium(expression(ctx.source))
-  case "CENTURY" | "CENTURIES" | "C" | "CENT" =>
-Century(expression(ctx.source))
-  case "DECADE" | "DECADES" | "DEC" | "DECS" =>
-Decade(expression(ctx.source))
-  case "YEAR" | "Y" | "YEARS" | "YR" | "YRS" =>
-Year(expression(ctx.source))
-  case "ISOYEAR" =>
-IsoYear(expression(ctx.source))
-  case "QUARTER" | "QTR" =>
-Quarter(expression(ctx.source))
-  case "MONTH" | "MON" | "MONS" | "MONTHS" =>
-Month(expression(ctx.source))
-  case "WEEK" | "W" | "WEEKS" =>
-WeekOfYear(expression(ctx.source))
-  case "DAY" | "D" | "DAYS" =>
-DayOfMonth(expression(ctx.source))
-  case "DAYOFWEEK" =>
-DayOfWeek(expression(ctx.source))
-  case "DOW" =>
-Subtract(DayOfWeek(expression(ctx.source)), Literal(1))
-  case "ISODOW" =>
-Add(WeekDay(expression(ctx.source)), Literal(1))
-  case "DOY" =>
-DayOfYear(expression(ctx.source))
-  case "HOUR" | "H" | "HOURS" | "HR" | "HRS" =>
-Hour(expression(ctx.source))
-  case "MINUTE" | "M" | "MIN" | "MINS" | "MINUTES" =>
-Minute(expression(ctx.source))
-  case "SECOND" | "S" | "SEC" | "SECONDS" | "SECS" =>
-Second(expression(ctx.source))
-  case "MILLISECONDS" | "MSEC" | "MSECS" | "MILLISECON" | "MSECONDS" | 
"MS" =>
-Milliseconds(expression(ctx.source))
-  case "MICROSECONDS" | "USEC" | "USECS" | "USECONDS" | "MICROSECON" | 
"US" =>
-Microseconds(expression(ctx.source))
-  case "EPOCH" =>
-Epoch(expression(ctx.source))
-  case other =>
-throw new ParseException(s"Literals of type '$other' are currently not 
supported.", ctx)
 
 Review comment:
   Ur, I don't think the approach is bad and I'm just looking for a better 
solution to keep the current error handling. If we already have the similar 
logic, can we follow that?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates

2019-08-17 Thread GitBox
maropu commented on a change in pull request #25410: [SPARK-28690][SQL] Add 
`date_part` function for timestamps/dates
URL: https://github.com/apache/spark/pull/25410#discussion_r314962673
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
 ##
 @@ -1409,48 +1409,7 @@ class AstBuilder(conf: SQLConf) extends 
SqlBaseBaseVisitor[AnyRef] with Logging
* Create a Extract expression.
*/
   override def visitExtract(ctx: ExtractContext): Expression = withOrigin(ctx) 
{
-ctx.field.getText.toUpperCase(Locale.ROOT) match {
-  case "MILLENNIUM" | "MILLENNIA" | "MIL" | "MILS" =>
-Millennium(expression(ctx.source))
-  case "CENTURY" | "CENTURIES" | "C" | "CENT" =>
-Century(expression(ctx.source))
-  case "DECADE" | "DECADES" | "DEC" | "DECS" =>
-Decade(expression(ctx.source))
-  case "YEAR" | "Y" | "YEARS" | "YR" | "YRS" =>
-Year(expression(ctx.source))
-  case "ISOYEAR" =>
-IsoYear(expression(ctx.source))
-  case "QUARTER" | "QTR" =>
-Quarter(expression(ctx.source))
-  case "MONTH" | "MON" | "MONS" | "MONTHS" =>
-Month(expression(ctx.source))
-  case "WEEK" | "W" | "WEEKS" =>
-WeekOfYear(expression(ctx.source))
-  case "DAY" | "D" | "DAYS" =>
-DayOfMonth(expression(ctx.source))
-  case "DAYOFWEEK" =>
-DayOfWeek(expression(ctx.source))
-  case "DOW" =>
-Subtract(DayOfWeek(expression(ctx.source)), Literal(1))
-  case "ISODOW" =>
-Add(WeekDay(expression(ctx.source)), Literal(1))
-  case "DOY" =>
-DayOfYear(expression(ctx.source))
-  case "HOUR" | "H" | "HOURS" | "HR" | "HRS" =>
-Hour(expression(ctx.source))
-  case "MINUTE" | "M" | "MIN" | "MINS" | "MINUTES" =>
-Minute(expression(ctx.source))
-  case "SECOND" | "S" | "SEC" | "SECONDS" | "SECS" =>
-Second(expression(ctx.source))
-  case "MILLISECONDS" | "MSEC" | "MSECS" | "MILLISECON" | "MSECONDS" | 
"MS" =>
-Milliseconds(expression(ctx.source))
-  case "MICROSECONDS" | "USEC" | "USECS" | "USECONDS" | "MICROSECON" | 
"US" =>
-Microseconds(expression(ctx.source))
-  case "EPOCH" =>
-Epoch(expression(ctx.source))
-  case other =>
-throw new ParseException(s"Literals of type '$other' are currently not 
supported.", ctx)
 
 Review comment:
   Ur, I don't think the approach is bad and I'm just looking for a better 
solution to keep the current error handling for that. If we already have the 
similar logic, can we follow that?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
SparkQA commented on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522277856
 
 
   **[Test build #109282 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109282/testReport)**
 for PR 22570 at commit 
[`24f95b7`](https://github.com/apache/spark/commit/24f95b72930fe2426c3f639115d7be88dec87925).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-17 Thread GitBox
maropu commented on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-522277764
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #25033: [SPARK-28224][SQL] Check overflow in decimal Sum aggregate

2019-08-17 Thread GitBox
maropu commented on a change in pull request #25033: [SPARK-28224][SQL] Check 
overflow in decimal Sum aggregate
URL: https://github.com/apache/spark/pull/25033#discussion_r314962435
 
 

 ##
 File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
 ##
 @@ -156,6 +156,33 @@ class DataFrameSuite extends QueryTest with 
SharedSQLContext {
   structDf.select(xxhash64($"a", $"record.*")))
   }
 
+  test("SPARK-28224: Aggregate sum big decimal overflow") {
+Seq(true, false).foreach { nullOnOverflow =>
+  withSQLConf((SQLConf.DECIMAL_OPERATIONS_NULL_ON_OVERFLOW.key, 
nullOnOverflow.toString)) {
+val structDf = largeDecimals.select("a").agg(sum("a"))
+if (nullOnOverflow) {
+  checkAnswer(structDf, Row(null))
+} else {
+  val e = intercept[Exception] {
+structDf.collect
+  }
+  assert(e.getCause.getClass.equals(classOf[ArithmeticException]))
+  assert(e.getCause.getMessage.contains("cannot be represented as 
Decimal"))
+}
+  }
+}
+  }
+
+  test("Aggregate sum integers") {
 
 Review comment:
   But, the case for int/double is an existing path in master, so we already 
have tests for that somewhere, right? I this the test above is enough for this 
pr. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source

2019-08-17 Thread GitBox
SparkQA commented on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS 
fail with Unable to move source
URL: https://github.com/apache/spark/pull/25364#issuecomment-522276526
 
 
   **[Test build #4836 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4836/testReport)**
 for PR 25364 at commit 
[`52421d1`](https://github.com/apache/spark/commit/52421d1850e3205d60d59e150b722f72f8a1feba).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source

2019-08-17 Thread GitBox
AmplabJenkins removed a comment on issue #25364: [SPARK-21067][DOC] Fix Thrift 
Server - CTAS fail with Unable to move source
URL: https://github.com/apache/spark/pull/25364#issuecomment-518487093
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] srowen commented on a change in pull request #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type

2019-08-17 Thread GitBox
srowen commented on a change in pull request #25480: [SPARK-28763][SQL][TEST] 
Flaky Tests: 
SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary 
type
URL: https://github.com/apache/spark/pull/25480#discussion_r314959107
 
 

 ##
 File path: 
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SparkThriftServerProtocolVersionsSuite.scala
 ##
 @@ -224,7 +224,9 @@ class SparkThriftServerProtocolVersionsSuite extends 
HiveThriftJdbcTest {
   }
   testExecuteStatementWithProtocolVersion(version, "SELECT cast(49960 as 
binary)") { rs =>
 assert(rs.next())
-assert(rs.getString(1) === 
UTF8String.fromBytes(NumberConverter.toBinary(49960)).toString)
+assertResult(Array(0, 0, 65533, 40).map(_.toChar)) {
+  rs.getString(1).toCharArray
+}
 
 Review comment:
   @shaneknapp I think we need to standardize `LANG` across the workers? or 
should I try giving it shot?
   That seems reasonable though I am distantly concerned that something in the 
code relies on this setting. That said, I think it's probably correct to set 
`LANG` in a prod environment and correct to set it to this value anyway.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



  1   2   3   4   >