[GitHub] [spark] AmplabJenkins removed a comment on issue #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
AmplabJenkins removed a comment on issue #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()` URL: https://github.com/apache/spark/pull/25408#issuecomment-520315935 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14025/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
AmplabJenkins removed a comment on issue #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()` URL: https://github.com/apache/spark/pull/25408#issuecomment-520315930 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
AmplabJenkins commented on issue #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()` URL: https://github.com/apache/spark/pull/25408#issuecomment-520315935 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14025/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
AmplabJenkins commented on issue #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()` URL: https://github.com/apache/spark/pull/25408#issuecomment-520315930 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
SparkQA commented on issue #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()` URL: https://github.com/apache/spark/pull/25408#issuecomment-520314941 **[Test build #108954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108954/testReport)** for PR 25408 at commit [`f025e6e`](https://github.com/apache/spark/commit/f025e6e38cca2f3e3f4522715e27ad7620fdd150). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
dongjoon-hyun commented on a change in pull request #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()` URL: https://github.com/apache/spark/pull/25408#discussion_r312793741 ## File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/date.sql ## @@ -228,9 +228,9 @@ SELECT f1 - date '2000-01-01' AS `Days From 2K` FROM DATE_TBL; -- test extract! -- -- epoch --- --- SELECT EXTRACT(EPOCH FROM DATE'1970-01-01'); -- 0 --- SELECT EXTRACT(EPOCH FROM TIMESTAMP '1970-01-01'); -- 0 Review comment: Thanks, @MaxGekk . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
dongjoon-hyun commented on a change in pull request #25408: [SPARK-28687][SQL]
Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
URL: https://github.com/apache/spark/pull/25408#discussion_r312793651
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##
@@ -1876,3 +1930,22 @@ case class Decade(child: Expression) extends
UnaryExpression with ImplicitCastIn
defineCodeGen(ctx, ev, c => s"$dtu.getDecade($c)")
}
}
+
+case class Epoch(child: Expression, timeZoneId: Option[String] = None)
+extends UnaryExpression with ImplicitCastInputTypes with
TimeZoneAwareExpression {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(TimestampType)
+ override def dataType: DataType = DecimalType(20, 6)
Review comment:
Could you add a comment about the above before line 1938, @MaxGekk ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25331: [SPARK-27768][SQL] Infinity, -Infinity, NaN should be recognized in a case insensitive manner.
dongjoon-hyun commented on issue #25331: [SPARK-27768][SQL] Infinity, -Infinity, NaN should be recognized in a case insensitive manner. URL: https://github.com/apache/spark/pull/25331#issuecomment-520314155 Sorry for being late, guys. Hi, @srowen . I agree with you that `Any improvement or bug fix is a behavior change`. However, this PR causes a different SQL result which can be considered as a correctness issue to the existing Spark SQL apps. For example, we can imagine a SQL reporting app which processes two rows `nan` and `NaN`. Previously `nan` is considered as `NULL`, but now `nan` become a `NaN`. So, when you do `GROUP BY` on this values, Spark 3.0 will return one row while Spark 2.4 returns two rows. For the infinity values, the situation is the same. In this case, we had better have a SQL migration guide. Otherwise, during Spark migration acceptance tests, SQL result comparison will fail because this kind of corner cases are tested always. Also, cc @gatorsmile since he filed this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs
AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs URL: https://github.com/apache/spark/pull/25368#issuecomment-520312099 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108946/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
MaxGekk commented on a change in pull request #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()` URL: https://github.com/apache/spark/pull/25408#discussion_r312791386 ## File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/date.sql ## @@ -228,9 +228,9 @@ SELECT f1 - date '2000-01-01' AS `Days From 2K` FROM DATE_TBL; -- test extract! -- -- epoch --- --- SELECT EXTRACT(EPOCH FROM DATE'1970-01-01'); -- 0 --- SELECT EXTRACT(EPOCH FROM TIMESTAMP '1970-01-01'); -- 0 Review comment: I can revert the comments back and check if it matters. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
MaxGekk commented on a change in pull request #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()` URL: https://github.com/apache/spark/pull/25408#discussion_r312791386 ## File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/date.sql ## @@ -228,9 +228,9 @@ SELECT f1 - date '2000-01-01' AS `Days From 2K` FROM DATE_TBL; -- test extract! -- -- epoch --- --- SELECT EXTRACT(EPOCH FROM DATE'1970-01-01'); -- 0 --- SELECT EXTRACT(EPOCH FROM TIMESTAMP '1970-01-01'); -- 0 Review comment: I can revert the comments back and check if it is matter. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs
AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs URL: https://github.com/apache/spark/pull/25368#issuecomment-520312097 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs
AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs URL: https://github.com/apache/spark/pull/25368#issuecomment-520312097 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs
AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs URL: https://github.com/apache/spark/pull/25368#issuecomment-520312099 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108946/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs
SparkQA removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs URL: https://github.com/apache/spark/pull/25368#issuecomment-520287517 **[Test build #108946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108946/testReport)** for PR 25368 at commit [`b07790d`](https://github.com/apache/spark/commit/b07790d133346d24ef92695ae1a61ad755e988cf). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
MaxGekk commented on a change in pull request #25408: [SPARK-28687][SQL]
Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
URL: https://github.com/apache/spark/pull/25408#discussion_r312791079
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##
@@ -1876,3 +1930,22 @@ case class Decade(child: Expression) extends
UnaryExpression with ImplicitCastIn
defineCodeGen(ctx, ev, c => s"$dtu.getDecade($c)")
}
}
+
+case class Epoch(child: Expression, timeZoneId: Option[String] = None)
+extends UnaryExpression with ImplicitCastInputTypes with
TimeZoneAwareExpression {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(TimestampType)
+ override def dataType: DataType = DecimalType(20, 6)
Review comment:
I tried `DoubleType` but I was loosing 1 microsecond in some tests.
Regarding `precision` (20) and `scale` (6). 6 was taken to have microseconds,
20 to cover supported range `[1, ]` of years + additional numbers to cover
negative years that are out of the range ( but it would be nice to cover them).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs
SparkQA commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs URL: https://github.com/apache/spark/pull/25368#issuecomment-520311718 **[Test build #108946 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108946/testReport)** for PR 25368 at commit [`b07790d`](https://github.com/apache/spark/commit/b07790d133346d24ef92695ae1a61ad755e988cf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #25406: [SPARK-28243][PYSPARK][ML][FOLLOW-UP] Move Python DecisionTreeParams to regression.py
huaxingao commented on issue #25406: [SPARK-28243][PYSPARK][ML][FOLLOW-UP] Move Python DecisionTreeParams to regression.py URL: https://github.com/apache/spark/pull/25406#issuecomment-520311578 @zhengruifeng This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] PavithraRamachandran commented on a change in pull request #25394: [SPARK-28671][SQL]when a non exsistent permanent function is dropped, NoSuchPermanentFunctionException is thrown
PavithraRamachandran commented on a change in pull request #25394:
[SPARK-28671][SQL]when a non exsistent permanent function is
dropped,NoSuchPermanentFunctionException is thrown
URL: https://github.com/apache/spark/pull/25394#discussion_r312788912
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##
@@ -1114,7 +1114,7 @@ class SessionCatalog(
}
externalCatalog.dropFunction(db, name.funcName)
} else if (!ignoreIfNotExists) {
- throw new NoSuchFunctionException(db = db, func = identifier.toString)
+ throw new NoSuchPermanentFunctionException(db = db, func =
identifier.toString)
Review comment:
**SPARK-14384** has removed the support of alter function., Later on alter
function was added as part of **create or replace** function support
(**SPARK-20383**). So when we use create or replace, the same issue will not
come as it creates a new a function if function does not exist.
There is no other place with such an issue
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] PavithraRamachandran commented on a change in pull request #25394: [SPARK-28671][SQL]when a non exsistent permanent function is dropped, NoSuchPermanentFunctionException is thrown
PavithraRamachandran commented on a change in pull request #25394:
[SPARK-28671][SQL]when a non exsistent permanent function is
dropped,NoSuchPermanentFunctionException is thrown
URL: https://github.com/apache/spark/pull/25394#discussion_r312788912
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##
@@ -1114,7 +1114,7 @@ class SessionCatalog(
}
externalCatalog.dropFunction(db, name.funcName)
} else if (!ignoreIfNotExists) {
- throw new NoSuchFunctionException(db = db, func = identifier.toString)
+ throw new NoSuchPermanentFunctionException(db = db, func =
identifier.toString)
Review comment:
**SPARK-14384** has removed the support of alter function., Later on alter
function was added as part of **create or replace** function support
(**SPARK-20383**). So when we use create or replace, the same issue will not
come as it creates a new a function if function does not exist.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
dongjoon-hyun commented on a change in pull request #25408: [SPARK-28687][SQL]
Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
URL: https://github.com/apache/spark/pull/25408#discussion_r312789139
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##
@@ -455,6 +456,22 @@ object DateTimeUtils {
(MICROSECONDS.toSeconds(localTimestamp(microsec, timeZone)) % 60).toInt
}
+ /**
+ * Returns seconds, including fractional parts, multiplied by 1000. The
timestamp
Review comment:
Yep. I'll put more details next time, @MaxGekk ~
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates
dongjoon-hyun commented on a change in pull request #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates URL: https://github.com/apache/spark/pull/25410#discussion_r312788922 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ## @@ -1876,3 +1876,38 @@ case class Decade(child: Expression) extends UnaryExpression with ImplicitCastIn defineCodeGen(ctx, ev, c => s"$dtu.getDecade($c)") } } + +case class DatePart(field: Expression, source: Expression, child: Expression) Review comment: Of course, you can run SQL module test always when you are writing SQL PRs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] PavithraRamachandran commented on a change in pull request #25394: [SPARK-28671][SQL]when a non exsistent permanent function is dropped, NoSuchPermanentFunctionException is thrown
PavithraRamachandran commented on a change in pull request #25394:
[SPARK-28671][SQL]when a non exsistent permanent function is
dropped,NoSuchPermanentFunctionException is thrown
URL: https://github.com/apache/spark/pull/25394#discussion_r312788912
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##
@@ -1114,7 +1114,7 @@ class SessionCatalog(
}
externalCatalog.dropFunction(db, name.funcName)
} else if (!ignoreIfNotExists) {
- throw new NoSuchFunctionException(db = db, func = identifier.toString)
+ throw new NoSuchPermanentFunctionException(db = db, func =
identifier.toString)
Review comment:
**SPARK-14384** has removed the support of alter function., Later on alter
function was added as part of **create or replace** function support
(**SPARK-20383**). So when we use create or replace, the same issue will not
come as it creates a new a function if function dies not exist.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates
dongjoon-hyun commented on a change in pull request #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates URL: https://github.com/apache/spark/pull/25410#discussion_r312788922 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ## @@ -1876,3 +1876,38 @@ case class Decade(child: Expression) extends UnaryExpression with ImplicitCastIn defineCodeGen(ctx, ev, c => s"$dtu.getDecade($c)") } } + +case class DatePart(field: Expression, source: Expression, child: Expression) Review comment: Of course, you can run SQL module test always when you are writing SQL PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #25408: [SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
MaxGekk commented on a change in pull request #25408: [SPARK-28687][SQL]
Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`
URL: https://github.com/apache/spark/pull/25408#discussion_r312788776
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##
@@ -455,6 +456,22 @@ object DateTimeUtils {
(MICROSECONDS.toSeconds(localTimestamp(microsec, timeZone)) % 60).toInt
}
+ /**
+ * Returns seconds, including fractional parts, multiplied by 1000. The
timestamp
Review comment:
I asked why it is better because @dongjoon-hyun suggested to changed without
explanation. Now it is clear for me, and I would keep this as is without gaps.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates
AmplabJenkins removed a comment on issue #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates URL: https://github.com/apache/spark/pull/25410#issuecomment-520309212 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base
AmplabJenkins removed a comment on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25415#issuecomment-520309158 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base
AmplabJenkins removed a comment on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25415#issuecomment-520309048 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates
AmplabJenkins removed a comment on issue #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates URL: https://github.com/apache/spark/pull/25410#issuecomment-520309215 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14024/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base
AmplabJenkins commented on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25415#issuecomment-520309524 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base
dongjoon-hyun commented on a change in pull request #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25415#discussion_r312788279 ## File path: sql/core/src/test/resources/sql-tests/inputs/udf/pgSQL/udf-select_having.sql ## @@ -6,7 +6,7 @@ -- https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/sql/select_having.sql -- -- This test file was converted from inputs/pgSQL/select_having.sql --- TODO: We should add UDFs in GROUP BY clause when [SPARK-28445] is resolved. +-- TODO: We should add UDFs in GROUP BY clause when [SPARK-28386] and [SPARK-26741] is resolved. Review comment: SPARK-28445 was wrong from the beginning, @shivusondur ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates
AmplabJenkins commented on issue #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates URL: https://github.com/apache/spark/pull/25410#issuecomment-520309215 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14024/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates
AmplabJenkins commented on issue #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates URL: https://github.com/apache/spark/pull/25410#issuecomment-520309212 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base
AmplabJenkins commented on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25415#issuecomment-520309158 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base
AmplabJenkins commented on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25415#issuecomment-520309048 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shivusondur commented on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base
shivusondur commented on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25415#issuecomment-520308548 @HyukjinKwon plz check or we can wait for corresponding jira([SPARK-28386] and [SPARK-26741]) to handle? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates
MaxGekk commented on a change in pull request #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates URL: https://github.com/apache/spark/pull/25410#discussion_r312787565 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ## @@ -1876,3 +1876,38 @@ case class Decade(child: Expression) extends UnaryExpression with ImplicitCastIn defineCodeGen(ctx, ev, c => s"$dtu.getDecade($c)") } } + +case class DatePart(field: Expression, source: Expression, child: Expression) Review comment: I added the description. Just in case, can I run the check locally and get the error earlier? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates
SparkQA commented on issue #25410: [SPARK-28690][SQL] Add the `date_part` function for timestamps and dates URL: https://github.com/apache/spark/pull/25410#issuecomment-520308253 **[Test build #108953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108953/testReport)** for PR 25410 at commit [`e68611a`](https://github.com/apache/spark/commit/e68611a5543adfae274549188e0e9945c4c399a3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shivusondur opened a new pull request #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base
shivusondur opened a new pull request #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP]Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25415 ## What changes were proposed in this pull request? only todo message updated. Need to add udf() for GroupBy Tests, after resolving following jira [SPARK-28386] and [SPARK-26741] ## How was this patch tested? NA, only TODO message updated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
dilipbiswal commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-520308207 @sandeep-katta Thanks for checking. So i think we are okay to do error checking up-front and make it consistent for both temp and permanent functions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25372: [SPARK-28640][SQL] Only give warning when session catalog is not defined
dongjoon-hyun commented on a change in pull request #25372: [SPARK-28640][SQL]
Only give warning when session catalog is not defined
URL: https://github.com/apache/spark/pull/25372#discussion_r312787057
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalog/v2/LookupCatalog.scala
##
@@ -62,6 +62,9 @@ trait LookupCatalog extends Logging {
try {
Some(lookupCatalog(SESSION_CATALOG_NAME))
} catch {
+ case _: CatalogNotFoundException =>
+logWarning("Session catalog is not defined")
+None
Review comment:
Could you fix both cases in this PR? I believe `typo` cases will be more
frequent than the empty configuration cases.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25414: [SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0 in VersionsSuite.scala on JDK 11
dongjoon-hyun commented on issue #25414: [SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0 in VersionsSuite.scala on JDK 11 URL: https://github.com/apache/spark/pull/25414#issuecomment-520307190 @wangyum . It's good to exclude that test, but the AS-IS comment seems inaccurate. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25414: [SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0 in VersionsSuite.scala on JDK 11
dongjoon-hyun commented on a change in pull request #25414:
[SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0
in VersionsSuite.scala on JDK 11
URL: https://github.com/apache/spark/pull/25414#discussion_r312786203
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala
##
@@ -620,6 +620,8 @@ class VersionsSuite extends SparkFunSuite with Logging {
test(s"$version: sql read hive materialized view") {
// HIVE-14249 Since Hive 2.3.0, materialized view is supported.
if (version == "2.3" || version == "3.0" || version == "3.1") {
+// Since Hive 3.0(HIVE-19383), client.runSqlHive is not supported on
JDK 11.
+assume(!(SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9) &&
version != "2.3"))
Review comment:
To claim the following, we should disable all `client.runSqlHive` in our
test suites.
> Since Hive 3.0(HIVE-19383), client.runSqlHive is not supported on JDK 11.
However, it seems that we have the followings.
```
$ git grep 'client.runSqlHive'
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogSuite.scala:
externalCatalog.client.runSqlHive(
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogSuite.scala:
val bucketString = externalCatalog.client.runSqlHive("DESC FORMATTED db1.t")
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala:
.client.runSqlHive(ddl)
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala:
client.runSqlHive("SET spark.sql.test.key=1")
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala:
client.runSqlHive("CREATE TABLE indexed_table (key INT)")
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala:
client.runSqlHive("CREATE INDEX index_1 ON TABLE indexed_table(key) " +
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala:
client.runSqlHive("CREATE TABLE materialized_view_tbl (c1 INT)")
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala:
client.runSqlHive(
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala:
client.runSqlHive(
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala:
client.runSqlHive(s"DROP INDEX IF EXISTS $indexName ON $tabName")
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala:
client.runSqlHive(
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala:
client.runSqlHive(
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala:
client.runSqlHive(
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala:
client.runSqlHive(s"ALTER TABLE $db.t ADD COLUMNS (dummy string)")
```
Could you be more specific?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sandeep-katta commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found
sandeep-katta commented on issue #25399: [SPARK-28670][SQL] create function should thrown Exception if the resource is not found URL: https://github.com/apache/spark/pull/25399#issuecomment-520306332 **Hive** In both (temporary and permanent) the cases query execution fails **case i: temporary function** jdbc:hive2://vm1:21066/> create temporary function addm AS 'com.huawei.bigdata.hive.example.udf.AddDoublesUDF' using jar 'hdfs://hacluster/user/AddDoublesUDF1.jar'; INFO : Executing command(queryId=omm_20190812133851_e58dd117-e8b1-40b6-8659-5ad14eddfdd6): create temporary function addm AS 'com.huawei.bigdata.hive.example.udf.AddDoublesUDF' using jar 'hdfs://hacluster/user/AddDoublesUDF1.jar'--0; Current sessionId=dcdc1020-3f73-4af2-95db-e834abda2020 **ERROR : File does not exist: hdfs://hacluster/user/AddDoublesUDF1.jar** Error: Error while processing statement: FAILED: Execution Error, return code -101 from **org.apache.hadoop.hive.ql.exec.FunctionTask. java.io.FileNotFoundException:** File does not exist: hdfs://hacluster/user/AddDoublesUDF1.jar (state=08S01,code=-101) **case ii: Permanent function** jdbc:hive2://vm1:21066/> create function addm AS 'com.huawei.bigdata.hive.example.udf.AddDoublesUDF' using jar 'hdfs://hacluster/user/AddDoublesUDF1.jar'; INFO : Executing command(queryId=omm_20190812133902_54e39039-b678-493e-93c2-8c09ce5bcfc0): create function addm AS 'com.huawei.bigdata.hive.example.udf.AddDoublesUDF' using jar 'hdfs://hacluster/user/AddDoublesUDF1.jar'--0; Current sessionId=dcdc1020-3f73-4af2-95db-e834abda2020 INFO : Starting task [Stage-0:FUNC] in serial mode **ERROR : File does not exist: hdfs://hacluster/user/AddDoublesUDF1.jar** ERROR : Failed to register default.addm using class com.huawei.bigdata.hive.example.udf.AddDoublesUDF Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask (state=08S01,code=1) **Presto:** As per presto there is no such temporary or permanent function concept, user needs to implement the UDF as plugin and put in the plugin folder and restart presto server. details about presto UDF are [here](https://www.qubole.com/blog/plugging-in-presto-udfs/) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on issue #25331: [SPARK-27768][SQL] Infinity, -Infinity, NaN should be recognized in a case insensitive manner.
dilipbiswal commented on issue #25331: [SPARK-27768][SQL] Infinity, -Infinity, NaN should be recognized in a case insensitive manner. URL: https://github.com/apache/spark/pull/25331#issuecomment-520306306 gentle ping @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24936: [SPARK-24634][SS] Add a new metric regarding number of rows later than watermark
SparkQA commented on issue #24936: [SPARK-24634][SS] Add a new metric regarding number of rows later than watermark URL: https://github.com/apache/spark/pull/24936#issuecomment-520305959 **[Test build #108952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108952/testReport)** for PR 24936 at commit [`54d159f`](https://github.com/apache/spark/commit/54d159fec0203f4edf615c8fd552df6c1f0b604f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24936: [SPARK-24634][SS] Add a new metric regarding number of rows later than watermark
AmplabJenkins removed a comment on issue #24936: [SPARK-24634][SS] Add a new metric regarding number of rows later than watermark URL: https://github.com/apache/spark/pull/24936#issuecomment-510987092 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-520304855 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108948/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-520304851 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
SparkQA removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-520294055 **[Test build #108948 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108948/testReport)** for PR 25201 at commit [`ac87ffc`](https://github.com/apache/spark/commit/ac87ffc95e0f95c147a725396447518811472c8a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
HyukjinKwon closed pull request #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor URL: https://github.com/apache/spark/pull/25411 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
AmplabJenkins commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-520304855 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108948/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
AmplabJenkins commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-520304851 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
SparkQA commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-520304737 **[Test build #108948 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108948/testReport)** for PR 25201 at commit [`ac87ffc`](https://github.com/apache/spark/commit/ac87ffc95e0f95c147a725396447518811472c8a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jzhuge commented on a change in pull request #25372: [SPARK-28640][SQL] Only give warning when session catalog is not defined
jzhuge commented on a change in pull request #25372: [SPARK-28640][SQL] Only
give warning when session catalog is not defined
URL: https://github.com/apache/spark/pull/25372#discussion_r312784188
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalog/v2/LookupCatalog.scala
##
@@ -62,6 +62,9 @@ trait LookupCatalog extends Logging {
try {
Some(lookupCatalog(SESSION_CATALOG_NAME))
} catch {
+ case _: CatalogNotFoundException =>
+logWarning("Session catalog is not defined")
+None
Review comment:
@dongjoon-hyun Thanks for the review. Your command line is not the case I
tried to fix in the PR. In your case, the stack trace is helpful.
It seems that the current master has session catalog defined by default, so
here is the command line to reproduce my case:
```
$ bin/spark-shell --master 'local[*]' --conf spark.sql.catalog.session=
...
Spark context available as 'sc' (master = local[*], app id =
local-1565588237201).
Spark session available as 'spark'.
...
scala> spark.sessionState.analyzer.sessionCatalog
...
2019-08-11 22:37:24,216 ERROR [main] hive.HiveSessionStateBuilder$$anon$1
(Logging.scala:logError(94)) - Cannot load v2 session catalog
org.apache.spark.SparkException: Cannot find catalog plugin class for
catalog 'session':
at org.apache.spark.sql.catalog.v2.Catalogs.load(Catalogs.java:81)
...
res0: Option[org.apache.spark.sql.catalog.v2.CatalogPlugin] = None
```
Here the stack trace does not add more information. And I am concerned that
if any rule uses session catalog, we will see this long stack trace again and
again.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jzhuge commented on a change in pull request #25372: [SPARK-28640][SQL] Only give warning when session catalog is not defined
jzhuge commented on a change in pull request #25372: [SPARK-28640][SQL] Only
give warning when session catalog is not defined
URL: https://github.com/apache/spark/pull/25372#discussion_r312784188
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalog/v2/LookupCatalog.scala
##
@@ -62,6 +62,9 @@ trait LookupCatalog extends Logging {
try {
Some(lookupCatalog(SESSION_CATALOG_NAME))
} catch {
+ case _: CatalogNotFoundException =>
+logWarning("Session catalog is not defined")
+None
Review comment:
@dongjoon-hyun Thanks for the review. Your command line is not the case I
tried to fix in the PR. In your case, the stack trace is helpful.
It seems that the current master has session catalog defined by default, so
here is the command line to reproduce my case:
```
$ bin/spark-shell --master 'local[*]' --conf spark.sql.catalog.session=
...
Spark context available as 'sc' (master = local[*], app id =
local-1565588237201).
Spark session available as 'spark'.
...
scala> spark.sessionState.analyzer.sessionCatalog
...
2019-08-11 22:37:24,216 ERROR [main] hive.HiveSessionStateBuilder$$anon$1
(Logging.scala:logError(94)) - Cannot load v2 session catalog
org.apache.spark.SparkException: Cannot find catalog plugin class for
catalog 'session':
at org.apache.spark.sql.catalog.v2.Catalogs.load(Catalogs.java:81)
...
res0: Option[org.apache.spark.sql.catalog.v2.CatalogPlugin] = None
```
Here is the stack trace does not add more information. And I am concerned
that if any rule uses session catalog, we will see this long stack trace again
and again.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25414: [SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0 in VersionsSuite.scala on JDK 11
AmplabJenkins removed a comment on issue #25414: [SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0 in VersionsSuite.scala on JDK 11 URL: https://github.com/apache/spark/pull/25414#issuecomment-520303874 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14023/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25414: [SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0 in VersionsSuite.scala on JDK 11
AmplabJenkins removed a comment on issue #25414: [SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0 in VersionsSuite.scala on JDK 11 URL: https://github.com/apache/spark/pull/25414#issuecomment-520303872 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25414: [SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0 in VersionsSuite.scala on JDK 11
SparkQA commented on issue #25414: [SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0 in VersionsSuite.scala on JDK 11 URL: https://github.com/apache/spark/pull/25414#issuecomment-520304097 **[Test build #108951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108951/testReport)** for PR 25414 at commit [`978a21a`](https://github.com/apache/spark/commit/978a21af5250c31525a4ba2fab96175a1e93871b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
HyukjinKwon commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor URL: https://github.com/apache/spark/pull/25411#issuecomment-520303909 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25414: [SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0 in VersionsSuite.scala on JDK 11
AmplabJenkins commented on issue #25414: [SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0 in VersionsSuite.scala on JDK 11 URL: https://github.com/apache/spark/pull/25414#issuecomment-520303872 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25414: [SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0 in VersionsSuite.scala on JDK 11
AmplabJenkins commented on issue #25414: [SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0 in VersionsSuite.scala on JDK 11 URL: https://github.com/apache/spark/pull/25414#issuecomment-520303874 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14023/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25405: [SPARK-28685][SQL][TEST] Test HMS 2.0.0+ in VersionsSuite/HiveClientSuites on JDK 11
HyukjinKwon commented on issue #25405: [SPARK-28685][SQL][TEST] Test HMS 2.0.0+ in VersionsSuite/HiveClientSuites on JDK 11 URL: https://github.com/apache/spark/pull/25405#issuecomment-520303800 @shaneknapp .. actually, adding `test-java11` in PR builder turned to be a bit important now.. @wangyum, mind filing an JIRA? For some context as FYI, for JDK 11 support, I asked feasibility about Hive 2.3.6 release at [this Hive dev thread](http://mail-archives.apache.org/mod_mbox/hive-dev/201908.mbox/%3CCANQiJeV3VM0iVp%2BgTwKPpx9dHeXe0BcLAicmWU5EPDOptb%2B_%2BQ%40mail.gmail.com%3E). Thankfully, seems positive. @wangyum quickly started worked on that and @alanfgates (from Hive) is actively cooperative - thanks again. Therefore, two communities are cooperating at [SPARK-28684](https://issues.apache.org/jira/browse/SPARK-28684) and [HIVE-22096](https://issues.apache.org/jira/browse/HIVE-22096) . It might be much easier if we can test JDK 11 in PR builder in Spark side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum opened a new pull request #25414: [SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0 in VersionsSuite.scala on JDK 11
wangyum opened a new pull request #25414: [SPARK-28688][SQL][TEST] Skip test `read hive materialized view` since Hive 3.0 in VersionsSuite.scala on JDK 11 URL: https://github.com/apache/spark/pull/25414 ## What changes were proposed in this pull request? This PR makes it skip test `read hive materialized view` since Hive 3.0 in `VersionsSuite.scala` on JDK 11 because [HIVE-19383](https://issues.apache.org/jira/browse/HIVE-19383) added [ArrayList$SubList](https://github.com/apache/hive/blob/ae4df627952610dbec029b099f0964908b3a4f25/ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java#L383) which is incompatible with JDK 11: ```java java.lang.RuntimeException: java.lang.NoSuchFieldException: parentOffset at org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:389) at org.apache.hadoop.hive.ql.exec.SerializationUtilities$1.create(SerializationUtilities.java:235) ... ```   ## How was this patch tested? manual tests **Test on JDK 11**: ``` ... [info] - 2.3: sql read hive materialized view (1 second, 253 milliseconds) ... [info] - 3.0: sql read hive materialized view !!! CANCELED !!! (31 milliseconds) [info] org.apache.commons.lang3.SystemUtils.isJavaVersionAtLeast(JAVA_9) was true, and "[3.0]" did not equal "[2.3]" (VersionsSuite.scala:624) ... [info] - 3.1: sql read hive materialized view !!! CANCELED !!! (1 millisecond) [info] org.apache.commons.lang3.SystemUtils.isJavaVersionAtLeast(JAVA_9) was true, and "[3.1]" did not equal "[2.3]" (VersionsSuite.scala:624) ... ``` **Test on JDK 1.8**: ``` ... [info] - 2.3: sql read hive materialized view (1 second, 444 milliseconds) ... [info] - 3.0: sql read hive materialized view (3 seconds, 100 milliseconds) ... [info] - 3.1: sql read hive materialized view (2 seconds, 941 milliseconds) ... ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
AmplabJenkins removed a comment on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor URL: https://github.com/apache/spark/pull/25411#issuecomment-520301172 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14022/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
AmplabJenkins removed a comment on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor URL: https://github.com/apache/spark/pull/25411#issuecomment-520301169 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
SparkQA commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor URL: https://github.com/apache/spark/pull/25411#issuecomment-520301164 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/14022/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
AmplabJenkins commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor URL: https://github.com/apache/spark/pull/25411#issuecomment-520301169 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
AmplabJenkins commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor URL: https://github.com/apache/spark/pull/25411#issuecomment-520301172 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14022/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-520299294 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108943/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-520299290 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-520299294 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108943/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-520299290 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
SparkQA removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-520279181 **[Test build #108943 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108943/testReport)** for PR 25333 at commit [`757491e`](https://github.com/apache/spark/commit/757491e3433fcd68d852abcfff26dfe07a2e07f4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
SparkQA commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor URL: https://github.com/apache/spark/pull/25411#issuecomment-520299074 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/14022/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
SparkQA commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-520299105 **[Test build #108943 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108943/testReport)** for PR 25333 at commit [`757491e`](https://github.com/apache/spark/commit/757491e3433fcd68d852abcfff26dfe07a2e07f4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
AmplabJenkins removed a comment on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor URL: https://github.com/apache/spark/pull/25411#issuecomment-520298813 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
AmplabJenkins removed a comment on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor URL: https://github.com/apache/spark/pull/25411#issuecomment-520298817 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108950/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
AmplabJenkins commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor URL: https://github.com/apache/spark/pull/25411#issuecomment-520298817 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108950/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
SparkQA removed a comment on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor URL: https://github.com/apache/spark/pull/25411#issuecomment-520297350 **[Test build #108950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108950/testReport)** for PR 25411 at commit [`bbc2e70`](https://github.com/apache/spark/commit/bbc2e708844fbcb18eacbfc404d75f67f17818d3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
AmplabJenkins commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor URL: https://github.com/apache/spark/pull/25411#issuecomment-520298813 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
SparkQA commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor URL: https://github.com/apache/spark/pull/25411#issuecomment-520298780 **[Test build #108950 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108950/testReport)** for PR 25411 at commit [`bbc2e70`](https://github.com/apache/spark/commit/bbc2e708844fbcb18eacbfc404d75f67f17818d3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25360: [SPARK-28280][PYTHON][SQL][TESTS][FOLLOW-UP] Add UDF cases into group by clause in 'udf-group-by.sql'
HyukjinKwon commented on a change in pull request #25360: [SPARK-28280][PYTHON][SQL][TESTS][FOLLOW-UP] Add UDF cases into group by clause in 'udf-group-by.sql' URL: https://github.com/apache/spark/pull/25360#discussion_r312779019 ## File path: sql/core/src/test/resources/sql-tests/inputs/udf/udf-group-by.sql ## @@ -20,29 +20,25 @@ SELECT 'foo', COUNT(udf(a)) FROM testData GROUP BY 1; SELECT 'foo' FROM testData WHERE a = 0 GROUP BY udf(1); -- Aggregate grouped by literals (hash aggregate). -SELECT 'foo', udf(APPROX_COUNT_DISTINCT(udf(a))) FROM testData WHERE a = 0 GROUP BY 1; +SELECT 'foo', udf(APPROX_COUNT_DISTINCT(udf(a))) FROM testData WHERE a = 0 GROUP BY udf(1); -- Aggregate grouped by literals (sort aggregate). -SELECT 'foo', MAX(STRUCT(udf(a))) FROM testData WHERE a = 0 GROUP BY 1; +SELECT 'foo', MAX(STRUCT(udf(a))) FROM testData WHERE a = 0 GROUP BY udf(1); -- Aggregate with complex GroupBy expressions. SELECT udf(a + b), udf(COUNT(b)) FROM testData GROUP BY a + b; SELECT udf(a + 2), udf(COUNT(b)) FROM testData GROUP BY a + 1; - --- [SPARK-28445] Inconsistency between Scala and Python/Panda udfs when groupby with udf() is used --- The following query will make Scala UDF work, but Python and Pandas udfs will fail with an AnalysisException. --- The query should be added after SPARK-28445. --- SELECT udf(a + 1), udf(COUNT(b)) FROM testData GROUP BY udf(a + 1); +SELECT udf(a + 1), udf(COUNT(b)) FROM testData GROUP BY udf(a + 1); Review comment: @skonto, looks all fine except this one. Let's fix and I'll get this in. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base
HyukjinKwon commented on a change in pull request #25371:
[SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF
test base
URL: https://github.com/apache/spark/pull/25371#discussion_r312778802
##
File path: sql/core/src/test/resources/sql-tests/inputs/udf/pgSQL/udf-join.sql
##
@@ -0,0 +1,2081 @@
+--
+-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+--
+--
+-- JOIN
+-- Test JOIN clauses
+--
https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/sql/join.sql
+--
+-- This test file was converted from pgSQL/join.sql.
+
+CREATE OR REPLACE TEMPORARY VIEW INT4_TBL AS SELECT * FROM
+ (VALUES (0), (123456), (-123456), (2147483647), (-2147483647))
+ AS v(f1);
+CREATE OR REPLACE TEMPORARY VIEW INT8_TBL AS SELECT * FROM
+ (VALUES
+(123, 456),
+(123, 4567890123456789),
+(4567890123456789, 123),
+(4567890123456789, 4567890123456789),
+(4567890123456789, -4567890123456789))
+ AS v(q1, q2);
+CREATE OR REPLACE TEMPORARY VIEW FLOAT8_TBL AS SELECT * FROM
+ (VALUES (0.0), (1004.30), (-34.84),
+(cast('1.2345678901234e+200' as double)), (cast('1.2345678901234e-200' as
double)))
+ AS v(f1);
+CREATE OR REPLACE TEMPORARY VIEW TEXT_TBL AS SELECT * FROM
+ (VALUES ('doh!'), ('hi de ho neighbor'))
+ AS v(f1);
+CREATE OR REPLACE TEMPORARY VIEW tenk2 AS SELECT * FROM tenk1;
+
+CREATE TABLE J1_TBL (
+ i integer,
+ j integer,
+ t string
+) USING parquet;
+
+CREATE TABLE J2_TBL (
+ i integer,
+ k integer
+) USING parquet;
+
+
+INSERT INTO J1_TBL VALUES (1, 4, 'one');
+INSERT INTO J1_TBL VALUES (2, 3, 'two');
+INSERT INTO J1_TBL VALUES (3, 2, 'three');
+INSERT INTO J1_TBL VALUES (4, 1, 'four');
+INSERT INTO J1_TBL VALUES (5, 0, 'five');
+INSERT INTO J1_TBL VALUES (6, 6, 'six');
+INSERT INTO J1_TBL VALUES (7, 7, 'seven');
+INSERT INTO J1_TBL VALUES (8, 8, 'eight');
+INSERT INTO J1_TBL VALUES (0, NULL, 'zero');
+INSERT INTO J1_TBL VALUES (NULL, NULL, 'null');
+INSERT INTO J1_TBL VALUES (NULL, 0, 'zero');
+
+INSERT INTO J2_TBL VALUES (1, -1);
+INSERT INTO J2_TBL VALUES (2, 2);
+INSERT INTO J2_TBL VALUES (3, -3);
+INSERT INTO J2_TBL VALUES (2, 4);
+INSERT INTO J2_TBL VALUES (5, -5);
+INSERT INTO J2_TBL VALUES (5, -5);
+INSERT INTO J2_TBL VALUES (0, NULL);
+INSERT INTO J2_TBL VALUES (NULL, NULL);
+INSERT INTO J2_TBL VALUES (NULL, 0);
+
+-- [SPARK-20856] Do not need onerow because it only used for test statement
using nested joins
+-- useful in some tests below
+-- create temp table onerow();
+-- insert into onerow default values;
+-- analyze onerow;
+
+
+--
+-- CORRELATION NAMES
+-- Make sure that table/column aliases are supported
+-- before diving into more complex join syntax.
+--
+
+SELECT udf('') AS `xxx`, udf(i), udf(j), udf(t)
Review comment:
@huaxingao, seems this file almost added `udf(...)` once for every
possibility. Can we use other combinations in general? For instance,
```
udf(...)
udf(udf(...))
```
or
```
on (udf(...) = ...)
on (... = udf(...))
on (udf(...) = udf(udf(...)))
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone
AmplabJenkins removed a comment on issue #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone URL: https://github.com/apache/spark/pull/25409#issuecomment-520297336 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108945/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone
AmplabJenkins removed a comment on issue #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone URL: https://github.com/apache/spark/pull/25409#issuecomment-520297334 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone
AmplabJenkins commented on issue #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone URL: https://github.com/apache/spark/pull/25409#issuecomment-520297334 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone
AmplabJenkins commented on issue #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone URL: https://github.com/apache/spark/pull/25409#issuecomment-520297336 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108945/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
SparkQA commented on issue #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor URL: https://github.com/apache/spark/pull/25411#issuecomment-520297350 **[Test build #108950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108950/testReport)** for PR 25411 at commit [`bbc2e70`](https://github.com/apache/spark/commit/bbc2e708844fbcb18eacbfc404d75f67f17818d3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone
SparkQA removed a comment on issue #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone URL: https://github.com/apache/spark/pull/25409#issuecomment-520283851 **[Test build #108945 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108945/testReport)** for PR 25409 at commit [`fbb4382`](https://github.com/apache/spark/commit/fbb4382d282be5e299e64da88e8ead3bd1d45292). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #25411: [SPARK-28652][TESTS][K8S] Add python version check for executor
viirya commented on a change in pull request #25411: [SPARK-28652][TESTS][K8S]
Add python version check for executor
URL: https://github.com/apache/spark/pull/25411#discussion_r312778434
##
File path: resource-managers/kubernetes/integration-tests/tests/pyfiles.py
##
@@ -35,4 +36,11 @@
# Begin of Python container checks
version_check(sys.argv[1], 2 if sys.argv[1] == "python" else 3)
+# Check python executable at executors
+spark.catalog.registerFunction("getSysVer",
+ lambda: "%d.%d" % sys.version_info[:2],
StringType())
+[row] = spark.sql("SELECT getSysVer()").collect()
+driverVersion = "%d.%d" % sys.version_info[:2]
Review comment:
Yes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone
SparkQA commented on issue #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone URL: https://github.com/apache/spark/pull/25409#issuecomment-520297134 **[Test build #108945 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108945/testReport)** for PR 25409 at commit [`fbb4382`](https://github.com/apache/spark/commit/fbb4382d282be5e299e64da88e8ead3bd1d45292). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #25404: [SPARK-28683][BUILD] Upgrade Scala to 2.12.9
dongjoon-hyun edited a comment on issue #25404: [SPARK-28683][BUILD] Upgrade Scala to 2.12.9 URL: https://github.com/apache/spark/pull/25404#issuecomment-520296446 Hi, All. Let's reuse SPARK-28683 for 2.12.10. I reopened the JIRA issue for that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25404: [SPARK-28683][BUILD] Upgrade Scala to 2.12.9
dongjoon-hyun commented on issue #25404: [SPARK-28683][BUILD] Upgrade Scala to 2.12.9 URL: https://github.com/apache/spark/pull/25404#issuecomment-520296446 Hi, All. Let's reuse SPARK-28683 for 2.12.10. I reopened it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
imback82 commented on a change in pull request #25247: [SPARK-28319][SQL]
Implement SHOW TABLES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25247#discussion_r312777236
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2SQLSuite.scala
##
@@ -1700,6 +1704,126 @@ class DataSourceV2SQLSuite extends QueryTest with
SharedSQLContext with BeforeAn
}
}
+ test("ShowTables: using v2 catalog") {
+spark.sql("CREATE TABLE testcat.db.table_name (id bigint, data string)
USING foo")
+spark.sql("CREATE TABLE testcat.n1.n2.db.table_name (id bigint, data
string) USING foo")
+
+runShowTablesSql("SHOW TABLES FROM testcat.db", Seq(Row("db",
"table_name")))
+
+runShowTablesSql(
+ "SHOW TABLES FROM testcat.n1.n2.db",
+ Seq(Row("n1.n2.db", "table_name")))
+ }
+
+ test("ShowTables: using v2 catalog with a pattern") {
+spark.sql("CREATE TABLE testcat.db.table (id bigint, data string) USING
foo")
+spark.sql("CREATE TABLE testcat.db.table_name_1 (id bigint, data string)
USING foo")
+spark.sql("CREATE TABLE testcat.db.table_name_2 (id bigint, data string)
USING foo")
+spark.sql("CREATE TABLE testcat.db2.table_name_2 (id bigint, data string)
USING foo")
+
+runShowTablesSql(
+ "SHOW TABLES FROM testcat.db",
+ Seq(
+Row("db", "table"),
+Row("db", "table_name_1"),
+Row("db", "table_name_2")))
+
+runShowTablesSql(
+ "SHOW TABLES FROM testcat.db LIKE '*name*'",
+ Seq(Row("db", "table_name_1"), Row("db", "table_name_2")))
+
+runShowTablesSql(
+ "SHOW TABLES FROM testcat.db LIKE '*2'",
+ Seq(Row("db", "table_name_2")))
+ }
+
+ test("ShowTables: using v2 catalog, namespace doesn't exist") {
+runShowTablesSql("SHOW TABLES FROM testcat.unknown", Seq())
Review comment:
@cloud-fan as far as I understand, throwing `NoSuchNamespaceException` is
optional in v2:
```
/**
* List the tables in a namespace from the catalog.
*
* If the catalog supports views, this must return identifiers for only
tables and not views.
*
* @param namespace a multi-part namespace
* @return an array of Identifiers for tables
* @throws NoSuchNamespaceException If the namespace does not exist
(optional).
*/
Identifier[] listTables(String[] namespace) throws
NoSuchNamespaceException;
```
I can update `TestInMemoryTableCatalog` to throw NoSuchNamespaceException if
there is no namespace existing for the tables created. However, I am not sure
if this is the right approach since you could have created namespace without
tables - in v1, you could have done `CREATE DATABASE db` without creating
tables belonging to `db`, although I don't think this scenario is supported in
v2 yet.
Please advise how this needs to be handled. Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25149: [SPARK-28383][SQL] SHOW CREATE TABLE is not supported on a temporary view
AmplabJenkins removed a comment on issue #25149: [SPARK-28383][SQL] SHOW CREATE TABLE is not supported on a temporary view URL: https://github.com/apache/spark/pull/25149#issuecomment-520295435 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14021/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25149: [SPARK-28383][SQL] SHOW CREATE TABLE is not supported on a temporary view
AmplabJenkins removed a comment on issue #25149: [SPARK-28383][SQL] SHOW CREATE TABLE is not supported on a temporary view URL: https://github.com/apache/spark/pull/25149#issuecomment-520295432 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24601: [SPARK-27702][K8S] Allow using some alternatives for service accounts
dongjoon-hyun commented on issue #24601: [SPARK-27702][K8S] Allow using some alternatives for service accounts URL: https://github.com/apache/spark/pull/24601#issuecomment-520295526 Sorry for being later, @Udbhav30 . If then, could you describe the test procedure in the `How was this patch tested?` section of PR description? For example, how to create new service account and how to use that? > i am unable to simulate this from minikube as there will always be a default service account so i am not sure if i could write any test case for this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
imback82 commented on a change in pull request #25247: [SPARK-28319][SQL]
Implement SHOW TABLES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25247#discussion_r312777236
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2SQLSuite.scala
##
@@ -1700,6 +1704,126 @@ class DataSourceV2SQLSuite extends QueryTest with
SharedSQLContext with BeforeAn
}
}
+ test("ShowTables: using v2 catalog") {
+spark.sql("CREATE TABLE testcat.db.table_name (id bigint, data string)
USING foo")
+spark.sql("CREATE TABLE testcat.n1.n2.db.table_name (id bigint, data
string) USING foo")
+
+runShowTablesSql("SHOW TABLES FROM testcat.db", Seq(Row("db",
"table_name")))
+
+runShowTablesSql(
+ "SHOW TABLES FROM testcat.n1.n2.db",
+ Seq(Row("n1.n2.db", "table_name")))
+ }
+
+ test("ShowTables: using v2 catalog with a pattern") {
+spark.sql("CREATE TABLE testcat.db.table (id bigint, data string) USING
foo")
+spark.sql("CREATE TABLE testcat.db.table_name_1 (id bigint, data string)
USING foo")
+spark.sql("CREATE TABLE testcat.db.table_name_2 (id bigint, data string)
USING foo")
+spark.sql("CREATE TABLE testcat.db2.table_name_2 (id bigint, data string)
USING foo")
+
+runShowTablesSql(
+ "SHOW TABLES FROM testcat.db",
+ Seq(
+Row("db", "table"),
+Row("db", "table_name_1"),
+Row("db", "table_name_2")))
+
+runShowTablesSql(
+ "SHOW TABLES FROM testcat.db LIKE '*name*'",
+ Seq(Row("db", "table_name_1"), Row("db", "table_name_2")))
+
+runShowTablesSql(
+ "SHOW TABLES FROM testcat.db LIKE '*2'",
+ Seq(Row("db", "table_name_2")))
+ }
+
+ test("ShowTables: using v2 catalog, namespace doesn't exist") {
+runShowTablesSql("SHOW TABLES FROM testcat.unknown", Seq())
Review comment:
@cloud-fan as far as I understand, throwing `NoSuchNamespaceException` is
optional:
```
/**
* List the tables in a namespace from the catalog.
*
* If the catalog supports views, this must return identifiers for only
tables and not views.
*
* @param namespace a multi-part namespace
* @return an array of Identifiers for tables
* @throws NoSuchNamespaceException If the namespace does not exist
(optional).
*/
Identifier[] listTables(String[] namespace) throws
NoSuchNamespaceException;
```
I can update `TestInMemoryTableCatalog` to throw NoSuchNamespaceException if
there is no namespace existing for the tables created. However, I am not sure
if this is the right approach since you could have created namespace without
tables - in v1, you could have done `CREATE DATABASE db` without creating
tables belonging to `db`, although I don't think this scenario is supported in
v2 yet.
Please advise how this needs to be handled. Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25149: [SPARK-28383][SQL] SHOW CREATE TABLE is not supported on a temporary view
AmplabJenkins commented on issue #25149: [SPARK-28383][SQL] SHOW CREATE TABLE is not supported on a temporary view URL: https://github.com/apache/spark/pull/25149#issuecomment-520295432 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
