[GitHub] spark pull request: [SPARK-2097][SQL] UDF Support
Github user DanielMe commented on the pull request: https://github.com/apache/spark/pull/1063#issuecomment-92678825 Okay, thanks for the clarification. Initially, I had naively assumed that the functionality you added was just a layer above the Hive API hence it was a bit confusing that `SHOW FUNCTIONS` did not list the UDFs. For my usecase I can easily work around that limitation so it's not that big of a deal. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2097][SQL] UDF Support
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1063#issuecomment-92457478 The biggest reason for the divergence is this API is much lighter weight (you can define functions in a single line, inline with the rest of your program). We can certainly consider adding more support for function listing metadata in the future, but you are the first to ask for this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2097][SQL] UDF Support
Github user DanielMe commented on the pull request: https://github.com/apache/spark/pull/1063#issuecomment-92301932 Excuse my naive question, however, it seems that this does not use the regular Hive UDF API, right? (Like when I would run `hiveContext.sql(CREATE TEMPORARY FUNCTION [...])` ). Is there any particular reason for that? I noticed, that UDFs created using this mechanism won't show up in the `SHOW FUNCTIONS` list. Would it be difficult to achieve that? Also, the Hive API allows to add description strings to a UDF (which obviously only makes sense if you can use `DESCRIBE FUNCTION`). It would be nice if something similar would exists for UDFs defined over the spark interface. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2097][SQL] UDF Support
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1063#issuecomment-50975589 QA tests have started for PR 1063. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17781/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2097][SQL] UDF Support
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1063#issuecomment-50975896
QA results for PR 1063:br- This patch PASSES unit tests.br- This patch
merges cleanlybr- This patch adds the following public classes
(experimental):brtrait OverrideFunctionRegistry extends FunctionRegistry
{brclass SimpleFunctionRegistry extends FunctionRegistry {brprotected[sql]
trait UdfRegistration {brclass JavaSQLContext(val sqlContext: SQLContext)
extends FunctionRegistration {brcase class EvaluatePython(udf: PythonUDF,
child: LogicalPlan) extends logical.UnaryNode {brcase class
BatchPythonEvaluation(udf: PythonUDF, output: Seq[Attribute], child:
SparkPlan)brbrFor more information see test
ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/1/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2097][SQL] UDF Support
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1063#issuecomment-50976747
QA results for PR 1063:br- This patch FAILED unit tests.br- This patch
merges cleanlybr- This patch adds the following public classes
(experimental):brtrait OverrideFunctionRegistry extends FunctionRegistry
{brclass SimpleFunctionRegistry extends FunctionRegistry {brprotected[sql]
trait UDFRegistration {brclass JavaSQLContext(val sqlContext: SQLContext)
extends UDFRegistration {brcase class EvaluatePython(udf: PythonUDF, child:
LogicalPlan) extends logical.UnaryNode {brcase class
BatchPythonEvaluation(udf: PythonUDF, output: Seq[Attribute], child:
SparkPlan)brbrFor more information see test
ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17779/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2097][SQL] UDF Support
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1063#issuecomment-50977296
QA results for PR 1063:br- This patch PASSES unit tests.br- This patch
merges cleanlybr- This patch adds the following public classes
(experimental):brtrait OverrideFunctionRegistry extends FunctionRegistry
{brclass SimpleFunctionRegistry extends FunctionRegistry {brprotected[sql]
trait UDFRegistration {brclass JavaSQLContext(val sqlContext: SQLContext)
extends UDFRegistration {brcase class EvaluatePython(udf: PythonUDF, child:
LogicalPlan) extends logical.UnaryNode {brcase class
BatchPythonEvaluation(udf: PythonUDF, output: Seq[Attribute], child:
SparkPlan)brbrFor more information see test
ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17781/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2097][SQL] UDF Support
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1063#issuecomment-50977690 Thanks for looking this over! I've merged to master and 1.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2097][SQL] UDF Support
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1063 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
