----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29754/#review70380 -----------------------------------------------------------
Two high-level comments: * RandomTask is complex. Developers will need recipes or good docs. This can be done as a separate. Just an observation. * Too many "context" classes. In general, should re-evaluate how everything is named. I threw in some random suggestions for context renaming. I'm still catching up on the discussion with Julian, so I might have more after that. build.gradle <https://reviews.apache.org/r/29754/#comment115519> Remove -d64 here. Tests should work on 32 bit machines. samza-sql/src/main/java/org/apache/samza/sql/api/data/Tuple.java <https://reviews.apache.org/r/29754/#comment115520> This API is a little strange to me. I'm assuming that this is getting a field from the message, right? samza-sql/src/main/java/org/apache/samza/sql/api/operators/routing/OperatorRoutingContext.java <https://reviews.apache.org/r/29754/#comment115523> OperatorRouter? samza-sql/src/main/java/org/apache/samza/sql/api/task/RuntimeSystemContext.java <https://reviews.apache.org/r/29754/#comment115521> SqlMessageCollector? RoutingMessageCollector? samza-sql/src/main/java/org/apache/samza/sql/task/RoutableRuntimeContext.java <https://reviews.apache.org/r/29754/#comment115526> OperatorMessageCollector? samza-sql/src/main/java/org/apache/samza/sql/task/StoredRuntimeContext.java <https://reviews.apache.org/r/29754/#comment115527> StoreMessageCollector? - Chris Riccomini On Jan. 29, 2015, 7:32 p.m., Yi Pan (Data Infrastructure) wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/29754/ > ----------------------------------------------------------- > > (Updated Jan. 29, 2015, 7:32 p.m.) > > > Review request for samza, Chris Riccomini, Jay Kreps, Milinda Pathirage, > Navina Ramesh, and Naveen Somasundaram. > > > Bugs: SAMZA-482 > https://issues.apache.org/jira/browse/SAMZA-482 > > > Repository: samza > > > Description > ------- > > Overview of the proposal: > - 4 different types of operators implemented via 2 interface classes: > * stream-to-relation operators implementing TupleOperator interface and > generate Relation output. E.g. Window operators > * stream-to-stream operators implementing TupleOperator interface and > generate Tuple output. E.g. Partition operators > * relation-to-relation operators implementing RelationOperator interface > and generate Relation output. E.g. All relational algebra operators, s.t. > join, where, group-by, select, etc. > * relation-to-stream operators implementing RelationOperator interface and > generate Tuple output. E.g. Istream or Dstream operators > > Those operators are connected via the following two context interface classes: > - RuntimeSystemContext which provides a context interface for the operators > to send their output to > - OperatorRoutingContext which provides the connection interface between the > operators > > In the example, we have enabled two execution models via the above two > context classes: > - RoutableRuntimeContext that uses the routing information from a > OperatorRoutingContext and directly invoking the next operator when the > current operator send its output via the RoutableRuntimeContext > - StoredRuntimeContext that provides a storage for each operator's outputs > that stored the output when the current operator send its output via > StoredRuntimeContext. > > Then, it is up to the programmer to query the StoredRuntimeContext to get the > operator's output and proceed w/ the next steps > The first execution model allows the integration w/ future SQL parser and > planner to automatically run a task, while the second model allows a random > programmer to use the operators from the library in a random context. > > > 2nd draft of SQL operator API > - Simplified data object interface > * NOTE that the following two issues are still pending revision: i) nested > data struct / schema; ii) multiple schema in a single stream? > - Simplified operator constructor > * provided simple constructor of build-in operators w/o the need to use > spec object and factory > - Added runtime context to allow operators to send out their output > * this reduced the types of the operators to only two: RelationOperator > and TupleOperator > - Added optional routing context to connect operators to each other > - Added two examples: > * i) a fully specified application that generates operators via factory > and automatically executes the operators via routing context > * ii) a "random" application that construct operators on-fly and use the > runtime context to get the intermediate results back > > > Diffs > ----- > > build.gradle 7a40ad4ae916610186848c06c4577e7067de98ee > gradle/dependency-versions.gradle 44dd42603e93788562fd64c68312570cee71a2aa > samza-sql/src/main/java/org/apache/samza/sql/api/data/EntityName.java > PRE-CREATION > samza-sql/src/main/java/org/apache/samza/sql/api/data/Relation.java > PRE-CREATION > samza-sql/src/main/java/org/apache/samza/sql/api/data/Tuple.java > PRE-CREATION > samza-sql/src/main/java/org/apache/samza/sql/api/operators/Operator.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/api/operators/RelationOperator.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/api/operators/TupleOperator.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/api/operators/routing/OperatorRoutingContext.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/api/task/RuntimeSystemContext.java > PRE-CREATION > samza-sql/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/partition/PartitionOp.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java > PRE-CREATION > samza-sql/src/main/java/org/apache/samza/sql/operators/relation/Join.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/routing/SimpleRoutingContext.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/stream/InsertStream.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/window/WindowState.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/task/RoutableRuntimeContext.java > PRE-CREATION > samza-sql/src/main/java/org/apache/samza/sql/task/StoredRuntimeContext.java > PRE-CREATION > samza-sql/src/test/java/org/apache/samza/sql/task/RandomOperatorTask.java > PRE-CREATION > samza-sql/src/test/java/org/apache/samza/sql/task/StreamSqlTask.java > PRE-CREATION > settings.gradle 3a01fd66359b8c79954ae8f34eeaf4b2e3fdc0b4 > > Diff: https://reviews.apache.org/r/29754/diff/ > > > Testing > ------- > > run ./bin/check-all.sh passed > > > Thanks, > > Yi Pan (Data Infrastructure) > >
