[GitHub] spark pull request: SPARK-2863: [SQL] Add facilities for function-...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/2768#issuecomment-67376279 hey @willb, thanks for working on this and sorry for the delay. I've spent some time thinking about this, and I'm wondering if maybe the right interface to do this is by expanding the information we collect when creating UDFs. I've written up some thoughts here: [SPARK-4867](https://issues.apache.org/jira/browse/SPARK-4867) I'd love to hear your thoughts on that approach, and please let me know if you are interested in working on any of the implementation. Regarding this particular PR, I'm on a quest to keep the queue as small as possible to help prevent things falling through the cracks. I propose we close this issue for now and revisit it once we have a full implementation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2863: [SQL] Add facilities for function-...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2768 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2863: [SQL] Add facilities for function-...
Github user willb commented on the pull request: https://github.com/apache/spark/pull/2768#issuecomment-60934954 I've rebased this to fix merge conflicts. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2863: [SQL] Add facilities for function-...
Github user willb commented on the pull request: https://github.com/apache/spark/pull/2768#issuecomment-58893902 Two open questions, and the latter is more relevant: is requiring that actuals are casted to the types of formals too restrictive? Is it likely to lead to type-coercion rules oscillating? (Obviously, it should be possible to pass, e.g., a value of a narrow numeric type where a wider one is expected. But if all the type-coercion rules we can anticipate ultimately widen types or convert from strings into other values, then the rules will still make progress.) (cc @marmbrus) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2863: [SQL] Add facilities for function-...
GitHub user willb opened a pull request: https://github.com/apache/spark/pull/2768 SPARK-2863: [SQL] Add facilities for function-argument coercion This commit adds the `SignedFunction` trait and modifies the `Sqrt` expression class to use it for coercing its argument to `DoubleType`. `SignedFunction` represents a fixed-arity function whose arguments should be casted to particular types. Expression classes extending SignedFunction must provide `formalTypes`, a List of expected types for formal parameters, `actualParams`, a list of Expressions corresponding to actual parameters, and create, which creates an instance of that expression class from a list of expressions corresponding to actuals. The type parameter for SignedFunction should be the expression class extending it. See the Sqrt class for a concrete example. This trait (or one or several abstract classes extending this trait) could be exposed to code outside `sql` in the future. You can merge this pull request into a Git repository by running: $ git pull https://github.com/willb/spark spark-2863 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2768.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2768 commit 4f9517a2c11d13f439f3ed7ea447a4559f9e9088 Author: William Benton wi...@redhat.com Date: 2014-10-11T12:40:10Z Adds SignedFunction trait and type coercion rules SignedFunction represents a fixed-arity function whose arguments should be casted to particular types. Expression classes extending SignedFunction must provide `formalTypes`, a List of expected types for formal parameters, `actualParams`, a list of Expressions corresponding to actual parameters, and create, which creates an instance of that expression class from a list of expressions corresponding to actuals. The type parameter for SignedFunction should be the expression class extending it. See the Sqrt class for a concrete example. This trait (or one or several abstract classes extending this trait) could be exposed to code outside `sql` in the future. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2863: [SQL] Add facilities for function-...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2768#issuecomment-58748502 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2863: [SQL] Add facilities for function-...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2768#issuecomment-58748576 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21646/consoleFull) for PR 2768 at commit [`4f9517a`](https://github.com/apache/spark/commit/4f9517a2c11d13f439f3ed7ea447a4559f9e9088). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2863: [SQL] Add facilities for function-...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2768#issuecomment-58749850 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21646/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2863: [SQL] Add facilities for function-...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2768#issuecomment-58749849 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21646/consoleFull) for PR 2768 at commit [`4f9517a`](https://github.com/apache/spark/commit/4f9517a2c11d13f439f3ed7ea447a4559f9e9088). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class Sqrt(child: Expression) extends UnaryExpression with SignedFunction[Sqrt] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
