[GitHub] [spark] AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520120962 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520120963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108907/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
SparkQA removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520117701 **[Test build #108907 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108907/testReport)** for PR 25383 at commit [`9c01d4b`](https://github.com/apache/spark/commit/9c01d4bfb362088e61644645946b5d44a94df262). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520120962 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520120963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108907/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
SparkQA commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520120904 **[Test build #108907 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108907/testReport)** for PR 25383 at commit [`9c01d4b`](https://github.com/apache/spark/commit/9c01d4bfb362088e61644645946b5d44a94df262). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
AmplabJenkins removed a comment on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-520120608 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108905/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
AmplabJenkins commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-520120608 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108905/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
AmplabJenkins commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-520120607 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
SparkQA removed a comment on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-520116013 **[Test build #108905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108905/testReport)** for PR 25399 at commit [`88b0418`](https://github.com/apache/spark/commit/88b041858857bb09a53b54a604f73ed6cbc46903). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
AmplabJenkins removed a comment on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-520120607 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
SparkQA commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-520120575 **[Test build #108905 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108905/testReport)** for PR 25399 at commit [`88b0418`](https://github.com/apache/spark/commit/88b041858857bb09a53b54a604f73ed6cbc46903). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
SparkQA commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-520119726 **[Test build #108909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108909/testReport)** for PR 24829 at commit [`7fe3507`](https://github.com/apache/spark/commit/7fe3507ed16ead5ad92cf6caf86088afd14ec344). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-520119617 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13983/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-520119616 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-520119617 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13983/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
dilipbiswal commented on a change in pull request #25399: [SPARK-28670][SQL]
create function should thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#discussion_r312691081
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
##
@@ -75,9 +76,18 @@ case class CreateFunctionCommand(
override def run(sparkSession: SparkSession): Seq[Row] = {
val catalog = sparkSession.sessionState.catalog
val func = CatalogFunction(FunctionIdentifier(functionName, databaseName),
className, resources)
+// throws Exception if it does not find the resource
+def checkIfResourceExists = {
+ resources.foreach(
+resource =>
+ if (!Utils.isFileExists(resource.uri,
sparkSession.sparkContext.hadoopConfiguration)) {
+throw new AnalysisException(s" Could not find the resource
${resource.uri}")
+ }
+ )
+}
if (isTemp) {
// We first load resources and then put the builder in the function
registry.
- catalog.loadFunctionResources(resources)
+ checkIfResourceExists
Review comment:
Previously we used to load the function resources. We don't need to do that
any more ? Is it done some place else ? If so, we need to adjust the comment
above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-520119616 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
wangyum commented on a change in pull request #24829: [SPARK-27988][SQL][TEST]
Port AGGREGATES.sql [Part 3]
URL: https://github.com/apache/spark/pull/24829#discussion_r312691029
##
File path:
sql/core/src/test/resources/sql-tests/results/pgSQL/aggregates_part3.sql.out
##
@@ -0,0 +1,108 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 11
+
+
+-- !query 0
+create temporary view varchar_tbl as select * from values
+ ('a'),
+ ('A'),
+ ('1'),
+ ('2'),
+ ('3'),
+ (''),
+ ('c')
+ as varchar_tbl(f1)
+-- !query 0 schema
+struct<>
+-- !query 0 output
+
+
+
+-- !query 1
+select concat_ws(',', collect_list(a)) from (values(''),(''),(''))
g(a)
+-- !query 1 schema
+struct
+-- !query 1 output
+,,
+
+
+-- !query 2
+select concat_ws(',', collect_list(a)) from
(values(''),(null),(''),('')) g(a)
+-- !query 2 schema
+struct
+-- !query 2 output
+,,
+
+
+-- !query 3
+select concat_ws('AB', collect_list(a)) from
(values(null),(null),(''),('')) g(a)
+-- !query 3 schema
+struct
+-- !query 3 output
+AB
+
+
+-- !query 4
+select concat_ws(',', collect_list(a)) from (values(null),(null)) g(a)
+-- !query 4 schema
+struct
+-- !query 4 output
+
+
+
+-- !query 5
+select concat_ws(',', sort_array(array_distinct(collect_list(f1 from
varchar_tbl
+-- !query 5 schema
+struct
+-- !query 5 output
+,1,2,3,A,a,c
Review comment:
Revert it to original query.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
wangyum commented on a change in pull request #24829: [SPARK-27988][SQL][TEST]
Port AGGREGATES.sql [Part 3]
URL: https://github.com/apache/spark/pull/24829#discussion_r312691032
##
File path:
sql/core/src/test/resources/sql-tests/inputs/pgSQL/aggregates_part3.sql
##
@@ -0,0 +1,287 @@
+--
+-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+--
+--
+-- AGGREGATES [Part 3]
+--
https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/aggregates.sql#L352-L605
+
+create temporary view varchar_tbl as select * from values
+ ('a'),
+ ('A'),
+ ('1'),
+ ('2'),
+ ('3'),
+ (''),
+ -- ('cd'),
+ ('c')
+ as varchar_tbl(f1);
+
+-- We do not support inheritance tree, skip related tests.
+-- try it on an inheritance tree
+-- create table minmaxtest(f1 int);
+-- create table minmaxtest1() inherits (minmaxtest);
+-- create table minmaxtest2() inherits (minmaxtest);
+-- create table minmaxtest3() inherits (minmaxtest);
+-- create index minmaxtesti on minmaxtest(f1);
+-- create index minmaxtest1i on minmaxtest1(f1);
+-- create index minmaxtest2i on minmaxtest2(f1 desc);
+-- create index minmaxtest3i on minmaxtest3(f1) where f1 is not null;
+
+-- insert into minmaxtest values(11), (12);
+-- insert into minmaxtest1 values(13), (14);
+-- insert into minmaxtest2 values(15), (16);
+-- insert into minmaxtest3 values(17), (18);
+
+-- explain (costs off)
+-- select min(f1), max(f1) from minmaxtest;
+-- select min(f1), max(f1) from minmaxtest;
+
+-- DISTINCT doesn't do anything useful here, but it shouldn't fail
+-- explain (costs off)
+-- select distinct min(f1), max(f1) from minmaxtest;
+-- select distinct min(f1), max(f1) from minmaxtest;
+
+-- drop table minmaxtest cascade;
+
+-- [SPARK-9830] It is not allowed to use an aggregate function in the argument
of another aggregate function
+-- check for correct detection of nested-aggregate errors
+-- select max(min(unique1)) from tenk1;
+-- select (select max(min(unique1)) from int8_tbl) from tenk1;
+
+-- These tests only test the explain. Skip these tests.
+--
+-- Test removal of redundant GROUP BY columns
+--
+
+-- create temp table t1 (a int, b int, c int, d int, primary key (a, b));
+-- create temp table t2 (x int, y int, z int, primary key (x, y));
+-- create temp table t3 (a int, b int, c int, primary key(a, b) deferrable);
+
+-- Non-primary-key columns can be removed from GROUP BY
+-- explain (costs off) select * from t1 group by a,b,c,d;
+
+-- No removal can happen if the complete PK is not present in GROUP BY
+-- explain (costs off) select a,c from t1 group by a,c,d;
+
+-- Test removal across multiple relations
+-- explain (costs off) select *
+-- from t1 inner join t2 on t1.a = t2.x and t1.b = t2.y
+-- group by t1.a,t1.b,t1.c,t1.d,t2.x,t2.y,t2.z;
+
+-- Test case where t1 can be optimized but not t2
+-- explain (costs off) select t1.*,t2.x,t2.z
+-- from t1 inner join t2 on t1.a = t2.x and t1.b = t2.y
+-- group by t1.a,t1.b,t1.c,t1.d,t2.x,t2.z;
+
+-- Cannot optimize when PK is deferrable
+-- explain (costs off) select * from t3 group by a,b,c;
+
+-- drop table t1;
+-- drop table t2;
+-- drop table t3;
+
+-- [SPARK-27974] Add built-in Aggregate Function: array_agg
+--
+-- Test combinations of DISTINCT and/or ORDER BY
+--
+
+-- select array_agg(a order by b)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+-- select array_agg(a order by a)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+-- select array_agg(a order by a desc)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+-- select array_agg(b order by a desc)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+
+-- select array_agg(distinct a)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+-- select array_agg(distinct a order by a)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+-- select array_agg(distinct a order by a desc)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+-- select array_agg(distinct a order by a desc nulls last)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+
+-- Skip the test below because it requires 4 UDFs: aggf_trans, aggfns_trans,
aggfstr, and aggfns
+-- multi-arg aggs, strict/nonstrict, distinct/order by
+
+-- select aggfstr(a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c);
+-- select aggfns(a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c);
+
+-- select aggfstr(distinct a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c),
+--generate_series(1,3) i;
+-- select aggfns(distinct a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c),
+--generate_series(1,3) i;
+
+-- select aggfstr(distinct a,b,c order by b)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c),
+--generate_series(1,3) i;
+-- select aggfns(distinct a,b,c order by b)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz'))
[GitHub] [spark] wangyum commented on a change in pull request #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
wangyum commented on a change in pull request #24829: [SPARK-27988][SQL][TEST]
Port AGGREGATES.sql [Part 3]
URL: https://github.com/apache/spark/pull/24829#discussion_r312691022
##
File path:
sql/core/src/test/resources/sql-tests/inputs/pgSQL/aggregates_part3.sql
##
@@ -0,0 +1,287 @@
+--
+-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+--
+--
+-- AGGREGATES [Part 3]
+--
https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/aggregates.sql#L352-L605
+
+create temporary view varchar_tbl as select * from values
+ ('a'),
+ ('A'),
+ ('1'),
+ ('2'),
+ ('3'),
+ (''),
+ -- ('cd'),
+ ('c')
+ as varchar_tbl(f1);
+
+-- We do not support inheritance tree, skip related tests.
+-- try it on an inheritance tree
+-- create table minmaxtest(f1 int);
+-- create table minmaxtest1() inherits (minmaxtest);
+-- create table minmaxtest2() inherits (minmaxtest);
+-- create table minmaxtest3() inherits (minmaxtest);
+-- create index minmaxtesti on minmaxtest(f1);
+-- create index minmaxtest1i on minmaxtest1(f1);
+-- create index minmaxtest2i on minmaxtest2(f1 desc);
+-- create index minmaxtest3i on minmaxtest3(f1) where f1 is not null;
+
+-- insert into minmaxtest values(11), (12);
+-- insert into minmaxtest1 values(13), (14);
+-- insert into minmaxtest2 values(15), (16);
+-- insert into minmaxtest3 values(17), (18);
+
+-- explain (costs off)
+-- select min(f1), max(f1) from minmaxtest;
+-- select min(f1), max(f1) from minmaxtest;
+
+-- DISTINCT doesn't do anything useful here, but it shouldn't fail
+-- explain (costs off)
+-- select distinct min(f1), max(f1) from minmaxtest;
+-- select distinct min(f1), max(f1) from minmaxtest;
+
+-- drop table minmaxtest cascade;
+
+-- [SPARK-9830] It is not allowed to use an aggregate function in the argument
of another aggregate function
+-- check for correct detection of nested-aggregate errors
+-- select max(min(unique1)) from tenk1;
+-- select (select max(min(unique1)) from int8_tbl) from tenk1;
+
+-- These tests only test the explain. Skip these tests.
+--
+-- Test removal of redundant GROUP BY columns
+--
+
+-- create temp table t1 (a int, b int, c int, d int, primary key (a, b));
+-- create temp table t2 (x int, y int, z int, primary key (x, y));
+-- create temp table t3 (a int, b int, c int, primary key(a, b) deferrable);
+
+-- Non-primary-key columns can be removed from GROUP BY
+-- explain (costs off) select * from t1 group by a,b,c,d;
+
+-- No removal can happen if the complete PK is not present in GROUP BY
+-- explain (costs off) select a,c from t1 group by a,c,d;
+
+-- Test removal across multiple relations
+-- explain (costs off) select *
+-- from t1 inner join t2 on t1.a = t2.x and t1.b = t2.y
+-- group by t1.a,t1.b,t1.c,t1.d,t2.x,t2.y,t2.z;
+
+-- Test case where t1 can be optimized but not t2
+-- explain (costs off) select t1.*,t2.x,t2.z
+-- from t1 inner join t2 on t1.a = t2.x and t1.b = t2.y
+-- group by t1.a,t1.b,t1.c,t1.d,t2.x,t2.z;
+
+-- Cannot optimize when PK is deferrable
+-- explain (costs off) select * from t3 group by a,b,c;
+
+-- drop table t1;
+-- drop table t2;
+-- drop table t3;
+
+-- [SPARK-27974] Add built-in Aggregate Function: array_agg
+--
+-- Test combinations of DISTINCT and/or ORDER BY
+--
+
+-- select array_agg(a order by b)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+-- select array_agg(a order by a)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+-- select array_agg(a order by a desc)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+-- select array_agg(b order by a desc)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+
+-- select array_agg(distinct a)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+-- select array_agg(distinct a order by a)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+-- select array_agg(distinct a order by a desc)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+-- select array_agg(distinct a order by a desc nulls last)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+
+-- Skip the test below because it requires 4 UDFs: aggf_trans, aggfns_trans,
aggfstr, and aggfns
+-- multi-arg aggs, strict/nonstrict, distinct/order by
+
+-- select aggfstr(a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c);
+-- select aggfns(a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c);
+
+-- select aggfstr(distinct a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c),
+--generate_series(1,3) i;
+-- select aggfns(distinct a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c),
+--generate_series(1,3) i;
+
+-- select aggfstr(distinct a,b,c order by b)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c),
+--generate_series(1,3) i;
+-- select aggfns(distinct a,b,c order by b)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz'))
[GitHub] [spark] wangyum commented on a change in pull request #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
wangyum commented on a change in pull request #24829: [SPARK-27988][SQL][TEST]
Port AGGREGATES.sql [Part 3]
URL: https://github.com/apache/spark/pull/24829#discussion_r312691004
##
File path:
sql/core/src/test/resources/sql-tests/results/pgSQL/aggregates_part3.sql.out
##
@@ -0,0 +1,108 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 11
+
+
+-- !query 0
+create temporary view varchar_tbl as select * from values
+ ('a'),
+ ('A'),
+ ('1'),
+ ('2'),
+ ('3'),
+ (''),
+ ('c')
+ as varchar_tbl(f1)
+-- !query 0 schema
+struct<>
+-- !query 0 output
+
+
+
+-- !query 1
+select concat_ws(',', collect_list(a)) from (values(''),(''),(''))
g(a)
+-- !query 1 schema
+struct
+-- !query 1 output
+,,
+
+
+-- !query 2
+select concat_ws(',', collect_list(a)) from
(values(''),(null),(''),('')) g(a)
+-- !query 2 schema
+struct
+-- !query 2 output
+,,
+
+
+-- !query 3
+select concat_ws('AB', collect_list(a)) from
(values(null),(null),(''),('')) g(a)
+-- !query 3 schema
+struct
+-- !query 3 output
+AB
+
+
+-- !query 4
+select concat_ws(',', collect_list(a)) from (values(null),(null)) g(a)
+-- !query 4 schema
+struct
+-- !query 4 output
+
+
+
+-- !query 5
+select concat_ws(',', sort_array(array_distinct(collect_list(f1 from
varchar_tbl
+-- !query 5 schema
+struct
+-- !query 5 output
+,1,2,3,A,a,c
+
+
+-- !query 6
+select min(CASE WHEN unique1> 100 THEN unique1 END) from tenk1
+-- !query 6 schema
+struct 100) THEN unique1 END):int>
+-- !query 6 output
+101
+
+
+-- !query 7
+select sum(CASE WHEN ten > 0 THEN cast(1/ten as integer) END) from tenk1
+-- !query 7 schema
+struct 0) THEN CAST((1 div ten) AS INT) END):bigint>
+-- !query 7 output
+1000
Review comment:
Revert it to original query.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
wangyum commented on a change in pull request #24829: [SPARK-27988][SQL][TEST]
Port AGGREGATES.sql [Part 3]
URL: https://github.com/apache/spark/pull/24829#discussion_r312691009
##
File path:
sql/core/src/test/resources/sql-tests/inputs/pgSQL/aggregates_part3.sql
##
@@ -0,0 +1,287 @@
+--
+-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+--
+--
+-- AGGREGATES [Part 3]
+--
https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/aggregates.sql#L352-L605
+
+create temporary view varchar_tbl as select * from values
+ ('a'),
+ ('A'),
+ ('1'),
+ ('2'),
+ ('3'),
+ (''),
+ -- ('cd'),
+ ('c')
+ as varchar_tbl(f1);
+
+-- We do not support inheritance tree, skip related tests.
+-- try it on an inheritance tree
+-- create table minmaxtest(f1 int);
+-- create table minmaxtest1() inherits (minmaxtest);
+-- create table minmaxtest2() inherits (minmaxtest);
+-- create table minmaxtest3() inherits (minmaxtest);
+-- create index minmaxtesti on minmaxtest(f1);
+-- create index minmaxtest1i on minmaxtest1(f1);
+-- create index minmaxtest2i on minmaxtest2(f1 desc);
+-- create index minmaxtest3i on minmaxtest3(f1) where f1 is not null;
+
+-- insert into minmaxtest values(11), (12);
+-- insert into minmaxtest1 values(13), (14);
+-- insert into minmaxtest2 values(15), (16);
+-- insert into minmaxtest3 values(17), (18);
+
+-- explain (costs off)
+-- select min(f1), max(f1) from minmaxtest;
+-- select min(f1), max(f1) from minmaxtest;
+
+-- DISTINCT doesn't do anything useful here, but it shouldn't fail
+-- explain (costs off)
+-- select distinct min(f1), max(f1) from minmaxtest;
+-- select distinct min(f1), max(f1) from minmaxtest;
+
+-- drop table minmaxtest cascade;
+
+-- [SPARK-9830] It is not allowed to use an aggregate function in the argument
of another aggregate function
+-- check for correct detection of nested-aggregate errors
+-- select max(min(unique1)) from tenk1;
+-- select (select max(min(unique1)) from int8_tbl) from tenk1;
+
+-- These tests only test the explain. Skip these tests.
+--
+-- Test removal of redundant GROUP BY columns
+--
+
+-- create temp table t1 (a int, b int, c int, d int, primary key (a, b));
+-- create temp table t2 (x int, y int, z int, primary key (x, y));
+-- create temp table t3 (a int, b int, c int, primary key(a, b) deferrable);
+
+-- Non-primary-key columns can be removed from GROUP BY
+-- explain (costs off) select * from t1 group by a,b,c,d;
+
+-- No removal can happen if the complete PK is not present in GROUP BY
+-- explain (costs off) select a,c from t1 group by a,c,d;
+
+-- Test removal across multiple relations
+-- explain (costs off) select *
+-- from t1 inner join t2 on t1.a = t2.x and t1.b = t2.y
+-- group by t1.a,t1.b,t1.c,t1.d,t2.x,t2.y,t2.z;
+
+-- Test case where t1 can be optimized but not t2
+-- explain (costs off) select t1.*,t2.x,t2.z
+-- from t1 inner join t2 on t1.a = t2.x and t1.b = t2.y
+-- group by t1.a,t1.b,t1.c,t1.d,t2.x,t2.z;
+
+-- Cannot optimize when PK is deferrable
+-- explain (costs off) select * from t3 group by a,b,c;
+
+-- drop table t1;
+-- drop table t2;
+-- drop table t3;
+
+-- [SPARK-27974] Add built-in Aggregate Function: array_agg
+--
+-- Test combinations of DISTINCT and/or ORDER BY
+--
+
+-- select array_agg(a order by b)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+-- select array_agg(a order by a)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+-- select array_agg(a order by a desc)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+-- select array_agg(b order by a desc)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+
+-- select array_agg(distinct a)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+-- select array_agg(distinct a order by a)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+-- select array_agg(distinct a order by a desc)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+-- select array_agg(distinct a order by a desc nulls last)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+
+-- Skip the test below because it requires 4 UDFs: aggf_trans, aggfns_trans,
aggfstr, and aggfns
+-- multi-arg aggs, strict/nonstrict, distinct/order by
+
+-- select aggfstr(a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c);
+-- select aggfns(a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c);
+
+-- select aggfstr(distinct a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c),
+--generate_series(1,3) i;
+-- select aggfns(distinct a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c),
+--generate_series(1,3) i;
+
+-- select aggfstr(distinct a,b,c order by b)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c),
+--generate_series(1,3) i;
+-- select aggfns(distinct a,b,c order by b)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz'))
[GitHub] [spark] wangyum commented on a change in pull request #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
wangyum commented on a change in pull request #24829: [SPARK-27988][SQL][TEST]
Port AGGREGATES.sql [Part 3]
URL: https://github.com/apache/spark/pull/24829#discussion_r312690968
##
File path:
sql/core/src/test/resources/sql-tests/inputs/pgSQL/aggregates_part3.sql
##
@@ -0,0 +1,287 @@
+--
+-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+--
+--
+-- AGGREGATES [Part 3]
+--
https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/aggregates.sql#L352-L605
+
+create temporary view varchar_tbl as select * from values
+ ('a'),
+ ('A'),
+ ('1'),
+ ('2'),
+ ('3'),
+ (''),
+ -- ('cd'),
+ ('c')
+ as varchar_tbl(f1);
+
+-- We do not support inheritance tree, skip related tests.
+-- try it on an inheritance tree
+-- create table minmaxtest(f1 int);
+-- create table minmaxtest1() inherits (minmaxtest);
+-- create table minmaxtest2() inherits (minmaxtest);
+-- create table minmaxtest3() inherits (minmaxtest);
+-- create index minmaxtesti on minmaxtest(f1);
+-- create index minmaxtest1i on minmaxtest1(f1);
+-- create index minmaxtest2i on minmaxtest2(f1 desc);
+-- create index minmaxtest3i on minmaxtest3(f1) where f1 is not null;
+
+-- insert into minmaxtest values(11), (12);
+-- insert into minmaxtest1 values(13), (14);
+-- insert into minmaxtest2 values(15), (16);
+-- insert into minmaxtest3 values(17), (18);
+
+-- explain (costs off)
+-- select min(f1), max(f1) from minmaxtest;
+-- select min(f1), max(f1) from minmaxtest;
+
+-- DISTINCT doesn't do anything useful here, but it shouldn't fail
+-- explain (costs off)
+-- select distinct min(f1), max(f1) from minmaxtest;
+-- select distinct min(f1), max(f1) from minmaxtest;
+
+-- drop table minmaxtest cascade;
+
+-- [SPARK-9830] It is not allowed to use an aggregate function in the argument
of another aggregate function
+-- check for correct detection of nested-aggregate errors
+-- select max(min(unique1)) from tenk1;
+-- select (select max(min(unique1)) from int8_tbl) from tenk1;
+
+-- These tests only test the explain. Skip these tests.
+--
+-- Test removal of redundant GROUP BY columns
+--
+
+-- create temp table t1 (a int, b int, c int, d int, primary key (a, b));
+-- create temp table t2 (x int, y int, z int, primary key (x, y));
+-- create temp table t3 (a int, b int, c int, primary key(a, b) deferrable);
+
+-- Non-primary-key columns can be removed from GROUP BY
+-- explain (costs off) select * from t1 group by a,b,c,d;
+
+-- No removal can happen if the complete PK is not present in GROUP BY
+-- explain (costs off) select a,c from t1 group by a,c,d;
+
+-- Test removal across multiple relations
+-- explain (costs off) select *
+-- from t1 inner join t2 on t1.a = t2.x and t1.b = t2.y
+-- group by t1.a,t1.b,t1.c,t1.d,t2.x,t2.y,t2.z;
+
+-- Test case where t1 can be optimized but not t2
+-- explain (costs off) select t1.*,t2.x,t2.z
+-- from t1 inner join t2 on t1.a = t2.x and t1.b = t2.y
+-- group by t1.a,t1.b,t1.c,t1.d,t2.x,t2.z;
+
+-- Cannot optimize when PK is deferrable
+-- explain (costs off) select * from t3 group by a,b,c;
+
+-- drop table t1;
+-- drop table t2;
+-- drop table t3;
+
+-- [SPARK-27974] Add built-in Aggregate Function: array_agg
+--
+-- Test combinations of DISTINCT and/or ORDER BY
+--
+
+-- select array_agg(a order by b)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+-- select array_agg(a order by a)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+-- select array_agg(a order by a desc)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+-- select array_agg(b order by a desc)
+-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b);
+
+-- select array_agg(distinct a)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+-- select array_agg(distinct a order by a)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+-- select array_agg(distinct a order by a desc)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+-- select array_agg(distinct a order by a desc nulls last)
+-- from (values (1),(2),(1),(3),(null),(2)) v(a);
+
+-- Skip the test below because it requires 4 UDFs: aggf_trans, aggfns_trans,
aggfstr, and aggfns
+-- multi-arg aggs, strict/nonstrict, distinct/order by
+
+-- select aggfstr(a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c);
+-- select aggfns(a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c);
+
+-- select aggfstr(distinct a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c),
+--generate_series(1,3) i;
+-- select aggfns(distinct a,b,c)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c),
+--generate_series(1,3) i;
+
+-- select aggfstr(distinct a,b,c order by b)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c),
+--generate_series(1,3) i;
+-- select aggfns(distinct a,b,c order by b)
+-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz'))
[GitHub] [spark] zhengruifeng commented on issue #25046: [SPARK-28243][PYSPARK][ML] Remove setFeatureSubsetStrategy and setSubsamplingRate from Python TreeEnsembleParams
zhengruifeng commented on issue #25046: [SPARK-28243][PYSPARK][ML] Remove setFeatureSubsetStrategy and setSubsamplingRate from Python TreeEnsembleParams URL: https://github.com/apache/spark/pull/25046#issuecomment-520119051 @huaxingao @srowen @mgaido91 I agree that we should remove those setter from the py side. However, we should not directly touch `param/shared.py`, instead we have to modify `_shared_params_code_gen.py` and then run `python _shared_params_code_gen.py > shared.py`. This is caused by that the `_shared_params_code_gen.py` will automatic generate both the setter and the getter, while in the scala side, only getter is generated. And in the scala side, `DecisionTreeParams` is not placed in `shareParam.scala`. There are too many design conflicts between the class hierarchy of scala and py, it's too confusing that can not be maintained easily. Maybe it is time to re-org the py side to keep it in line the scala side. I found this when I'm adding `Implement Tree-Based Feature Transformation` https://github.com/apache/spark/pull/25383 in the py side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
AmplabJenkins removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-520118983 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108901/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
AmplabJenkins commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-520118983 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108901/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
AmplabJenkins commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-520118981 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
AmplabJenkins removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-520118981 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
SparkQA removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-520108428 **[Test build #108901 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108901/testReport)** for PR 24232 at commit [`96fb0ad`](https://github.com/apache/spark/commit/96fb0ad110fc0e0cc3e8f8097fd370143999a7c2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
SparkQA commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-520118907 **[Test build #108901 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108901/testReport)** for PR 24232 at commit [`96fb0ad`](https://github.com/apache/spark/commit/96fb0ad110fc0e0cc3e8f8097fd370143999a7c2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions
SparkQA commented on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions URL: https://github.com/apache/spark/pull/25395#issuecomment-520118504 **[Test build #108908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108908/testReport)** for PR 25395 at commit [`5a3ff4a`](https://github.com/apache/spark/commit/5a3ff4ae6b64b7b2e31242c68f0e9034c6a43a08). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] kiszk commented on a change in pull request #25370: [SPARK-28639][CORE][DOC] Configuration doc for Barrier Execution Mode
kiszk commented on a change in pull request #25370: [SPARK-28639][CORE][DOC] Configuration doc for Barrier Execution Mode URL: https://github.com/apache/spark/pull/25370#discussion_r312690538 ## File path: docs/configuration.md ## @@ -2018,6 +2018,50 @@ Apart from these, the following properties are also available, and may be useful +### Barrier Execution Mode + + +Property NameDefaultMeaning + + spark.barrier.sync.timeout + 365d + +The timeout in seconds for each barrier() call from a barrier task. If the +coordinator didn't receive all the sync messages from barrier tasks within the +configed time, throw a SparkException to fail all the tasks. The default value is set Review comment: Yes, I think so. This is because other sections use `configured`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions
AmplabJenkins removed a comment on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions URL: https://github.com/apache/spark/pull/25395#issuecomment-520118396 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions
AmplabJenkins removed a comment on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions URL: https://github.com/apache/spark/pull/25395#issuecomment-520118398 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13982/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions
AmplabJenkins commented on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions URL: https://github.com/apache/spark/pull/25395#issuecomment-520118398 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13982/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions
AmplabJenkins commented on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions URL: https://github.com/apache/spark/pull/25395#issuecomment-520118396 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520118119 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108904/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520118118 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520118119 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108904/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520118118 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
SparkQA removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520114845 **[Test build #108904 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108904/testReport)** for PR 25383 at commit [`f104995`](https://github.com/apache/spark/commit/f104995b666ad448cb88f7e8ce33b9f2a96b1a1a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
SparkQA commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520118059 **[Test build #108904 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108904/testReport)** for PR 25383 at commit [`f104995`](https://github.com/apache/spark/commit/f104995b666ad448cb88f7e8ce33b9f2a96b1a1a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
SparkQA commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520117701 **[Test build #108907 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108907/testReport)** for PR 25383 at commit [`9c01d4b`](https://github.com/apache/spark/commit/9c01d4bfb362088e61644645946b5d44a94df262). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520117603 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13981/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520117603 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13981/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520117602 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520117602 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create
function should thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#discussion_r312689938
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
##
@@ -88,6 +98,7 @@ case class CreateFunctionCommand(
// For a permanent, we will store the metadata into underlying
external catalog.
// This function will be loaded into the FunctionRegistry when a query
uses it.
// We do not load it into FunctionRegistry right now.
+checkIfResourceExists
Review comment:
How about this?
```
// Checks if the given resources exist
val hadoopConf = sparkSession.sparkContext.hadoopConfiguration
val nonExistentResources = resources.filter { r =>
val path = new Path(r.uri)
!path.getFileSystem(hadoopConf).exists(path)
}
if (nonExistentResources.nonEmpty) {
throw new AnalysisException(s"Resources not found:
${nonExistentResources.mkString(",")}")
}
```
btw, this check should be placed in the start of
`CreateFunctionCommand.run`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions
SparkQA commented on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions URL: https://github.com/apache/spark/pull/25395#issuecomment-520116402 **[Test build #108906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108906/testReport)** for PR 25395 at commit [`527042f`](https://github.com/apache/spark/commit/527042f4b8a52b0e0470b027eeb1e20eb11d6a79). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create
function should thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#discussion_r312689538
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##
@@ -3180,6 +3180,20 @@ class SQLQuerySuite extends QueryTest with
SharedSQLContext {
}
}
+
+ test("SPARK-28670 " +
+"create function should throw AnalysisException if the UDF class is not
found") {
+Seq("", "temporary").foreach(funcType => {
+ val exp = intercept[AnalysisException] {
+sql(
+ s"""
+ |CREATE ${funcType} FUNCTION udtf_test AS
'org.apache.spark.sql.hive.execution.UDFTest'
+ |USING JAR '/var/invalid/invalid.jar'
+""".stripMargin)
+ }
+ assert(exp.getMessage.contains("Could not find the resource"))
+})
Review comment:
nit: How about this?
```
Seq(true, false).foreach { isTemporary =>
val exp = intercept[AnalysisException] {
sql(
s"""
|CREATE ${if (isTemporary) "TEMPORARY" else ""} FUNCTION
udtf_test
|AS 'org.apache.spark.sql.hive.execution.UDFTest'
|USING JAR '/var/invalid/invalid.jar'
""".stripMargin)
}
assert(exp.getMessage.contains("Could not find the resource"))
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions
AmplabJenkins removed a comment on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions URL: https://github.com/apache/spark/pull/25395#issuecomment-520116305 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions
AmplabJenkins removed a comment on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions URL: https://github.com/apache/spark/pull/25395#issuecomment-520116307 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13980/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create
function should thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#discussion_r312689483
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##
@@ -3180,6 +3180,20 @@ class SQLQuerySuite extends QueryTest with
SharedSQLContext {
}
}
+
+ test("SPARK-28670 " +
+"create function should throw AnalysisException if the UDF class is not
found") {
Review comment:
nit `test("SPARK-28670: create function should throw AnalysisException if
UDF classes not found") {`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create
function should thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#discussion_r312689483
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##
@@ -3180,6 +3180,20 @@ class SQLQuerySuite extends QueryTest with
SharedSQLContext {
}
}
+
+ test("SPARK-28670 " +
+"create function should throw AnalysisException if the UDF class is not
found") {
Review comment:
nit `test("SPARK-28670 create function should throw AnalysisException if
UDF classes not found") {`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions
AmplabJenkins commented on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions URL: https://github.com/apache/spark/pull/25395#issuecomment-520116305 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions
AmplabJenkins commented on issue #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions URL: https://github.com/apache/spark/pull/25395#issuecomment-520116307 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13980/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create
function should thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#discussion_r312689423
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##
@@ -3180,6 +3180,20 @@ class SQLQuerySuite extends QueryTest with
SharedSQLContext {
}
}
+
+ test("SPARK-28670 " +
+"create function should throw AnalysisException if the UDF class is not
found") {
Review comment:
nit: ` test("SPARK-28670: throw AnalysisException if UDF classes not
found") {`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create
function should thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#discussion_r312689423
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##
@@ -3180,6 +3180,20 @@ class SQLQuerySuite extends QueryTest with
SharedSQLContext {
}
}
+
+ test("SPARK-28670 " +
+"create function should throw AnalysisException if the UDF class is not
found") {
Review comment:
nit: ` test("SPARK-28670: throw AnalysisException if UDF classes not
found") {`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
maropu commented on a change in pull request #25399: [SPARK-28670][SQL] create
function should thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#discussion_r312689423
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##
@@ -3180,6 +3180,20 @@ class SQLQuerySuite extends QueryTest with
SharedSQLContext {
}
}
+
+ test("SPARK-28670 " +
+"create function should throw AnalysisException if the UDF class is not
found") {
Review comment:
nit: ` test("SPARK-28670 create function should throw AnalysisException if
the UDF class is not found") {`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
SparkQA commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-520116013 **[Test build #108905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108905/testReport)** for PR 25399 at commit [`88b0418`](https://github.com/apache/spark/commit/88b041858857bb09a53b54a604f73ed6cbc46903). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
AmplabJenkins removed a comment on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-520115888 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13979/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
AmplabJenkins removed a comment on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-520115887 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
AmplabJenkins commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-520115888 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13979/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
AmplabJenkins commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-520115887 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
zhengruifeng commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520115803 @WeichenXu123 @mgaido91 Could you please help reviewing this too? This feature involve some complexity, but shoule be useful. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
AmplabJenkins removed a comment on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-519916346 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
maropu commented on issue #25399: [SPARK-28670][SQL] create function should
thrown Exception if the resource is not found
URL: https://github.com/apache/spark/pull/25399#issuecomment-520115657
Really? I got;
```
scala> sql("""CREATE temporary FUNCTION udtf_test AS
'org.apache.spark.sql.hive.execution.UDFTest' USING JAR
'/var/invalid/invalid.jar'""")
19/08/10 12:58:19 ERROR SparkContext: Failed to add /var/invalid/invalid.jar
to Spark environment
java.io.FileNotFoundException: Jar /var/invalid/invalid.jar not found
at
org.apache.spark.SparkContext.addLocalJarFile$1(SparkContext.scala:1798)
at org.apache.spark.SparkContext.addJar(SparkContext.scala:1850)
```
Anyway, could you update the PR description for the before/after behaivours?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
maropu commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-520115661 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
zhengruifeng commented on a change in pull request #25383: [SPARK-13677][ML]
Implement Tree-Based Feature Transformation for ML
URL: https://github.com/apache/spark/pull/25383#discussion_r312689248
##
File path:
mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala
##
@@ -210,6 +214,25 @@ class DecisionTreeClassificationModel private[ml] (
rootNode.predictImpl(features).prediction
}
+ /** @group setParam */
+ @Since("3.0.0")
+ def setLeafCol(value: String): this.type = set(leafCol, value)
+
+ @Since("3.0.0")
+ def predictLeaf(features: Vector): Double = predictLeafImpl(features)
Review comment:
@srowen I prefer not to move `setLeafCol` into the superclass, since it
looks like the mllib's convention, such as `setVarianceCol` in
`DecisionTreeRegressionModel`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520115169 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520115171 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13978/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520115169 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520115171 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13978/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
SparkQA commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-520114845 **[Test build #108904 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108904/testReport)** for PR 25383 at commit [`f104995`](https://github.com/apache/spark/commit/f104995b666ad448cb88f7e8ce33b9f2a96b1a1a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25394: [SPARK-28671][SQL]when a non exsistent permanent function is dropped, NoSuchPermanentFunctionException is thrown
AmplabJenkins removed a comment on issue #25394: [SPARK-28671][SQL]when a non exsistent permanent function is dropped,NoSuchPermanentFunctionException is thrown URL: https://github.com/apache/spark/pull/25394#issuecomment-520114500 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108900/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25394: [SPARK-28671][SQL]when a non exsistent permanent function is dropped, NoSuchPermanentFunctionException is thrown
AmplabJenkins commented on issue #25394: [SPARK-28671][SQL]when a non exsistent permanent function is dropped,NoSuchPermanentFunctionException is thrown URL: https://github.com/apache/spark/pull/25394#issuecomment-520114500 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108900/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25394: [SPARK-28671][SQL]when a non exsistent permanent function is dropped, NoSuchPermanentFunctionException is thrown
AmplabJenkins removed a comment on issue #25394: [SPARK-28671][SQL]when a non exsistent permanent function is dropped,NoSuchPermanentFunctionException is thrown URL: https://github.com/apache/spark/pull/25394#issuecomment-520114498 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25394: [SPARK-28671][SQL]when a non exsistent permanent function is dropped, NoSuchPermanentFunctionException is thrown
AmplabJenkins commented on issue #25394: [SPARK-28671][SQL]when a non exsistent permanent function is dropped,NoSuchPermanentFunctionException is thrown URL: https://github.com/apache/spark/pull/25394#issuecomment-520114498 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25394: [SPARK-28671][SQL]when a non exsistent permanent function is dropped, NoSuchPermanentFunctionException is thrown
SparkQA commented on issue #25394: [SPARK-28671][SQL]when a non exsistent permanent function is dropped,NoSuchPermanentFunctionException is thrown URL: https://github.com/apache/spark/pull/25394#issuecomment-520114409 **[Test build #108900 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108900/testReport)** for PR 25394 at commit [`234235c`](https://github.com/apache/spark/commit/234235ce92fb08598fbca3a2149ca27319375829). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25394: [SPARK-28671][SQL]when a non exsistent permanent function is dropped, NoSuchPermanentFunctionException is thrown
SparkQA removed a comment on issue #25394: [SPARK-28671][SQL]when a non exsistent permanent function is dropped,NoSuchPermanentFunctionException is thrown URL: https://github.com/apache/spark/pull/25394#issuecomment-520100364 **[Test build #108900 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108900/testReport)** for PR 25394 at commit [`234235c`](https://github.com/apache/spark/commit/234235ce92fb08598fbca3a2149ca27319375829). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
zhengruifeng commented on a change in pull request #25383: [SPARK-13677][ML]
Implement Tree-Based Feature Transformation for ML
URL: https://github.com/apache/spark/pull/25383#discussion_r312688458
##
File path:
mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala
##
@@ -220,17 +227,23 @@ class DecisionTreeRegressionModel private[ml] (
var predictionColumns = Seq.empty[Column]
if ($(predictionCol).nonEmpty) {
- val predictUDF = udf { (features: Vector) => predict(features) }
+ val predictUDF = udf { vector: Vector => predict(vector) }
Review comment:
to remove the unnecessary brackets, then I rename the var to keep in line
with other places. I am neutral to revert these.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25349: [SPARK-28538][UI][WIP] Document SQL page
AmplabJenkins removed a comment on issue #25349: [SPARK-28538][UI][WIP] Document SQL page URL: https://github.com/apache/spark/pull/25349#issuecomment-520113005 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108902/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25349: [SPARK-28538][UI][WIP] Document SQL page
AmplabJenkins removed a comment on issue #25349: [SPARK-28538][UI][WIP] Document SQL page URL: https://github.com/apache/spark/pull/25349#issuecomment-520113003 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25349: [SPARK-28538][UI][WIP] Document SQL page
SparkQA removed a comment on issue #25349: [SPARK-28538][UI][WIP] Document SQL page URL: https://github.com/apache/spark/pull/25349#issuecomment-520112370 **[Test build #108902 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108902/testReport)** for PR 25349 at commit [`6c17d74`](https://github.com/apache/spark/commit/6c17d742946be9f148af244542cc53bb39eeea60). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25349: [SPARK-28538][UI][WIP] Document SQL page
SparkQA commented on issue #25349: [SPARK-28538][UI][WIP] Document SQL page URL: https://github.com/apache/spark/pull/25349#issuecomment-520112992 **[Test build #108902 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108902/testReport)** for PR 25349 at commit [`6c17d74`](https://github.com/apache/spark/commit/6c17d742946be9f148af244542cc53bb39eeea60). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25349: [SPARK-28538][UI][WIP] Document SQL page
AmplabJenkins commented on issue #25349: [SPARK-28538][UI][WIP] Document SQL page URL: https://github.com/apache/spark/pull/25349#issuecomment-520113003 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25349: [SPARK-28538][UI][WIP] Document SQL page
AmplabJenkins commented on issue #25349: [SPARK-28538][UI][WIP] Document SQL page URL: https://github.com/apache/spark/pull/25349#issuecomment-520113005 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108902/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
SparkQA commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-520112821 **[Test build #108903 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108903/testReport)** for PR 24232 at commit [`47b100b`](https://github.com/apache/spark/commit/47b100b76ebcad1fa35862cb3b841f4d0c2a4431). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
AmplabJenkins removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-520112690 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13977/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
AmplabJenkins removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-520112687 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25349: [SPARK-28538][UI][WIP] Document SQL page
AmplabJenkins removed a comment on issue #25349: [SPARK-28538][UI][WIP] Document SQL page URL: https://github.com/apache/spark/pull/25349#issuecomment-520112677 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13976/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
AmplabJenkins commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-520112690 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13977/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25349: [SPARK-28538][UI][WIP] Document SQL page
AmplabJenkins commented on issue #25349: [SPARK-28538][UI][WIP] Document SQL page URL: https://github.com/apache/spark/pull/25349#issuecomment-520112677 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13976/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25349: [SPARK-28538][UI][WIP] Document SQL page
AmplabJenkins removed a comment on issue #25349: [SPARK-28538][UI][WIP] Document SQL page URL: https://github.com/apache/spark/pull/25349#issuecomment-520112676 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25349: [SPARK-28538][UI][WIP] Document SQL page
AmplabJenkins commented on issue #25349: [SPARK-28538][UI][WIP] Document SQL page URL: https://github.com/apache/spark/pull/25349#issuecomment-520112676 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
AmplabJenkins commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-520112687 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] nvander1 commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
nvander1 commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-520112443 I'm also adding tests for the overloads that accept Java Functional Interfaces. - ~transform~ - ~map_filter~ - ~filter~ - ~exists~ - forall - aggregate - map_zip_with - transform_keys - transform_values - zip_with This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25349: [SPARK-28538][UI][WIP] Document SQL page
SparkQA commented on issue #25349: [SPARK-28538][UI][WIP] Document SQL page URL: https://github.com/apache/spark/pull/25349#issuecomment-520112370 **[Test build #108902 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108902/testReport)** for PR 25349 at commit [`6c17d74`](https://github.com/apache/spark/commit/6c17d742946be9f148af244542cc53bb39eeea60). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sandeep-katta commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
sandeep-katta commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-520109686 for temporary function it throws AnaylysisException but permanent function will be created successfully  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
