GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/6693
[SPARK-8149][SQL] Break ExpressionEvaluationSuite down to multiple files
Also moved a few files in expressions package around to match test suites.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rxin/spark expr-refactoring
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6693.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6693
----
commit b127ff8a0c5fb704da574d101a2d0e27ac5f463a
Author: cody koeninger <[email protected]>
Date: 2015-06-07T20:42:45Z
[SPARK-2808] [STREAMING] [KAFKA] cleanup tests from
see if requiring producer acks eliminates the need for
waitUntilLeaderOffset calls in tests
Author: cody koeninger <[email protected]>
Closes #5921 from koeninger/kafka-0.8.2-test-cleanup and squashes the
following commits:
1e89dc8 [cody koeninger] Merge branch 'master' into kafka-0.8.2-test-cleanup
4662828 [cody koeninger] [Streaming][Kafka] filter mima issue for removal
of method from private test class
af1e083 [cody koeninger] Merge branch 'master' into kafka-0.8.2-test-cleanup
4298ac2 [cody koeninger] [Streaming][Kafka] update comment to trigger
jenkins attempt
1274afb [cody koeninger] [Streaming][Kafka] see if requiring producer acks
eliminates the need for waitUntilLeaderOffset calls in tests
commit 5e7b6b67bed9cd0d8c7d4e78df666b807e8f7ef2
Author: Davies Liu <[email protected]>
Date: 2015-06-07T21:11:20Z
[SPARK-8117] [SQL] Push codegen implementation into each Expression
This PR move codegen implementation of expressions into Expression class
itself, make it easy to manage.
It introduces two APIs in Expression:
```
def gen(ctx: CodeGenContext): GeneratedExpressionCode
def genCode(ctx: CodeGenContext, ev: GeneratedExpressionCode): Code
```
gen(ctx) will call genSource(ctx, ev) to generate Java source code for the
current expression. A expression needs to override genSource().
Here are the types:
```
type Term String
type Code String
/**
* Java source for evaluating an [[Expression]] given a [[Row]] of input.
*/
case class GeneratedExpressionCode(var code: Code,
nullTerm: Term,
primitiveTerm: Term,
objectTerm: Term)
/**
* A context for codegen, which is used to bookkeeping the expressions
those are not supported
* by codegen, then they are evaluated directly. The unsupported expression
is appended at the
* end of `references`, the position of it is kept in the code, used to
access and evaluate it.
*/
class CodeGenContext {
/**
* Holding all the expressions those do not support codegen, will be
evaluated directly.
*/
val references: Seq[Expression] = new mutable.ArrayBuffer[Expression]()
}
```
This is basically #6660, but fixed style violation and compilation failure.
Author: Davies Liu <[email protected]>
Author: Reynold Xin <[email protected]>
Closes #6690 from rxin/codegen and squashes the following commits:
e1368c2 [Reynold Xin] Fixed tests.
73db80e [Reynold Xin] Fixed compilation failure.
19d6435 [Reynold Xin] Fixed style violation.
9adaeaf [Davies Liu] address comments
f42c732 [Davies Liu] improve coverage and tests
bad6828 [Davies Liu] address comments
e03edaa [Davies Liu] consts fold
86fac2c [Davies Liu] fix style
02262c9 [Davies Liu] address comments
b5d3617 [Davies Liu] Merge pull request #5 from rxin/codegen
48c454f [Reynold Xin] Some code gen update.
2344bc0 [Davies Liu] fix test
12ff88a [Davies Liu] fix build
c5fb514 [Davies Liu] rename
8c6d82d [Davies Liu] update docs
b145047 [Davies Liu] fix style
e57959d [Davies Liu] add type alias
3ff25f8 [Davies Liu] refactor
593d617 [Davies Liu] pushing codegen into Expression
commit b3a40f8592aabebdb5dc5a9ab0cfcb8580aa6faf
Author: Reynold Xin <[email protected]>
Date: 2015-06-07T08:23:55Z
Refactored expression test suites.
commit c0eb74b84a9bc54f23ee785e32f65921773cfa39
Author: Reynold Xin <[email protected]>
Date: 2015-06-07T21:39:58Z
Fixed compilation.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
