[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/446#discussion_r12707068 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -114,13 +120,19 @@ case class Or(left: Expression, right: Expression) extends BinaryPredicate { override def eval(input: Row): Any = { val l = left.eval(input) -val r = right.eval(input) -if (l == true || r == true) { +if(l == true) { true -} else if (l == null || r == null) { - null } else { - false + val r = right.eval(input) + if(r == true) { --- End diff -- mind adding a space here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-43287508 LGTM. I will merge it once Jenkins comes back green. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/446#discussion_r12707082 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -133,8 +145,12 @@ case class Equals(left: Expression, right: Expression) extends BinaryComparison def symbol = = override def eval(input: Row): Any = { val l = left.eval(input) -val r = right.eval(input) -if (l == null || r == null) null else l == r +if (l == null) { + null +} else { + val r = right.eval(input) + if(r == null) null else l == r --- End diff -- mind adding a space here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-43287373 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-43298012 I've merged this. Thanks a lot! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/446 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/446#discussion_r12707097 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala --- @@ -248,17 +248,31 @@ private[hive] case class HiveGenericUdf(name: String, children: Seq[Expression]) isUDFDeterministic children.foldLeft(true)((prev, n) = prev n.foldable) } + protected lazy val deferedObjects = Array.fill[DeferredObject](children.length)({ +new DeferredObjectAdapter + }) + + // Adapter from Catalyst ExpressionResult to Hive DeferredObject + class DeferredObjectAdapter extends DeferredObject { +private var func: () = Any = _ +def set(func: () = Any) { + this.func = func +} +override def prepare(i: Int) = {} +override def get(): AnyRef = wrap(func()) + } + val dataType: DataType = inspectorToDataType(returnInspector) override def eval(input: Row): Any = { returnInspector // Make sure initialized. -val args = children.map { v = - new DeferredObject { -override def prepare(i: Int) = {} -override def get(): AnyRef = wrap(v.eval(input)) - } -}.toArray -unwrap(function.evaluate(args)) +var i = 0 +while(i children.length) { --- End diff -- mind adding a space here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42718956 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14856/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/446#discussion_r12707063 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -98,13 +98,19 @@ case class And(left: Expression, right: Expression) extends BinaryPredicate { override def eval(input: Row): Any = { val l = left.eval(input) -val r = right.eval(input) -if (l == false || r == false) { - false -} else if (l == null || r == null ) { - null +if(l == false) { + false } else { - true + val r = right.eval(input) + if(r == false) { --- End diff -- mind adding a space here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/446#discussion_r12707061 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -98,13 +98,19 @@ case class And(left: Expression, right: Expression) extends BinaryPredicate { override def eval(input: Row): Any = { val l = left.eval(input) -val r = right.eval(input) -if (l == false || r == false) { - false -} else if (l == null || r == null ) { - null +if(l == false) { --- End diff -- mind adding a space here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-43291258 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15039/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-43287365 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-43291256 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-43287344 Thanks @rxin. It's done, can you re-test it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42713596 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42720579 LGTM @pwendell can you please merge? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42713490 test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42805496 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42805513 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42805599 Rebase to the latest master, can you test it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42811549 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42811551 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14896/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42874850 @pwendell any idea what is wrong with MIMA? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42881251 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42875260 This was an issue with another PR that was merged while failing. Should be fixed in master now. Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42881275 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42890217 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42890218 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14905/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42881061 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42718954 Build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42713579 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-42751036 @chenghao-intel this has some merge conflicts - mind updating it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/446#discussion_r11813931 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -146,7 +158,7 @@ case class If(predicate: Expression, trueValue: Expression, falseValue: Expressi extends Expression { def children = predicate :: trueValue :: falseValue :: Nil - def nullable = trueValue.nullable || falseValue.nullable + override def nullable = predicate.nullable || (trueValue.nullable falseValue.nullable) --- End diff -- I think this was correct before. The nullability of the `predicate` does not affect the nullability of the output since a null predicate will just cause the `falseValue` to be output, not null. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40949662 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40949725 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40949715 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40958506 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40958509 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14285/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40960369 @andrewor14, any idea what is up with `org.apache.spark.ui.UISuite`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40963941 It has the same problem as `streaming.UISuite`, which was disabled recently. We should do the same for `ui.UISuite` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40992799 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40996418 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40996419 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14311/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40999182 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40999175 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-41002706 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-41002708 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14314/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40916043 @marmbrus , I've removed the unrelated changes from this PR, and this PR only for the deferred expression evaluation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
GitHub user chenghao-intel opened a pull request: https://github.com/apache/spark/pull/446 [Spark-1461] Deferred Expression Evaluation (short-circuit evaluation) Short-circut will significantly improves the performance in Expression Evaluation, however, we can not fold an deterministic-less UDF (like Rand()), and also the stateful UDF should not be ignored in short-circuit evaluation(e.g. in expression: col1 0 and row_sequence() 1000, row_sequence() can not be ignored.) I brought an concept of DeferredObject from Hive, which has 2 types of children types (EagerResult / DeferredResult), the former will require trigger the evaluation before it's created, while the later will trigger the evaluation when first call its get() method. You can merge this pull request into a Git repository by running: $ git pull https://github.com/chenghao-intel/spark expression_deferred_evaluation Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/446.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #446 commit 6e1e99b5f157e18a713717d9fd2f643fb2576f85 Author: Cheng Hao hao.ch...@intel.com Date: 2014-04-11T05:49:45Z unify the nullable/stateful/foldable interface commit 6f6776bb860d9cbf5d7a23de7ff2376f1123c384 Author: Cheng Hao hao.ch...@intel.com Date: 2014-04-11T07:31:40Z Append some missing types for HiveUDF commit 9d827c00e684084661bd763c3849ef81bae84023 Author: Cheng Hao hao.ch...@intel.com Date: 2014-04-15T05:34:42Z refactor the expression with deferred evaluation commit 5d9febfa610106201ae4c8703b7a01a5c4b21a0d Author: Cheng Hao hao.ch...@intel.com Date: 2014-04-16T03:25:35Z fix cann't be compiled issues commit ded73054e3c0d75b5c0df01c02c33e5816973191 Author: Cheng Hao hao.ch...@intel.com Date: 2014-04-16T07:38:45Z Fix bug that un-initialized Array of DeferredObjectAdatper commit 850aec31310aaf3aab4609b9fb8262c0ef688aa1 Author: Cheng Hao hao.ch...@intel.com Date: 2014-04-17T08:05:20Z Fix bug of ClassCastedException commit 53f1fb5343815a58d31ee5d0fb262586211c2a86 Author: Cheng Hao hao.ch...@intel.com Date: 2014-04-17T08:24:32Z rename the Expression.compute = Expression.deferCompute commit 14e16cf527b6f5aea487720b90c3a3d3ba4764a3 Author: Cheng Hao hao.ch...@intel.com Date: 2014-04-18T02:09:05Z using lazy val instead of val in Cast commit 640df9aab2939a5ff601deeccc87d04b11e0a4a7 Author: Cheng Hao hao.ch...@intel.com Date: 2014-04-18T06:01:10Z make the Expression.eval as public method commit 9b2ebbaca628291606ba9979c4566588c769403b Author: Cheng Hao hao.ch...@intel.com Date: 2014-04-18T06:03:52Z rename Expression.deferCompute = Expression.deferEval --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40790412 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40854164 A few high-level comments: - I'm not sure if stateful UDFs are actually something we want to support. The semantics for them are not well defined in partitioned systems, especially where the optimizer decides the partitioning. If you want things like row id there are already ways to do this with map partitions with index. - The deferred evaluation class seems like a complicated way to get short circuit evaluation. In a lot of cases can't we just change the ordering of calling the existing eval method? Adding a new interface complicates things, and in some simple benchmarks that I ran this code is actually slower than what was there before (probably because of the extra object allocations). - There are a lot of unrelated changes here also. While fixing a minor spelling error or something is okay, making a whole bunch of unrelated changes makes reviewing the PR more difficult for us. For example, maybe you can do the data type additions for Hive UDFs in their own PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/446#issuecomment-40859590 Thank you @marmbrus , if we are not planning to support the stateful UDFs, the deferred evaluation can be removed, too, and we can just change the ordering of calling the eval method for the existed expressions for short-circuit evaluation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---