[GitHub] incubator-gearpump pull request #85: [GEARPUMP-23] add window dsl
Github user asfgit closed the pull request at: https://github.com/apache/incubator-gearpump/pull/85 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-gearpump pull request #85: [GEARPUMP-23] add window dsl
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/85#discussion_r80809137
--- Diff:
streaming/src/main/scala/org/apache/gearpump/streaming/dsl/Stream.scala ---
@@ -147,6 +150,33 @@ class Stream[T](
graph.addEdge(thisNode, edge.getOrElse(Shuffle), processorOp)
new Stream[R](graph, processorOp, Some(Shuffle))
}
+
+ private def groupByWindow[GROUP](fn: GroupByFn[T, GROUP],
+ parallelism: Int, description: String): GroupByStream[T, GROUP] = {
+val groupOp = GroupByOp[T, GROUP](fn,
+ parallelism, description)
+graph.addVertex(groupOp)
+graph.addEdge(thisNode, edge.getOrElse(Shuffle), groupOp)
+new GroupByStream[T, GROUP](graph, groupOp)
+ }
+
+}
+
+class GroupByStream[T, GROUP](
+private val graph: Graph[Op, OpEdge],
+private val groupByOp: GroupByOp[T, GROUP],
+private val edge: Option[OpEdge] = None) extends Stream[T](graph,
groupByOp, edge) {
--- End diff --
`private val` is visible to companion object
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #85: [GEARPUMP-23] add window dsl
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/85#discussion_r80808708
--- Diff:
streaming/src/main/scala/org/apache/gearpump/streaming/dsl/Stream.scala ---
@@ -115,20 +121,17 @@ class Stream[T](
*
* For example,
* {{{
- * Stream[People].groupBy(_.gender).flatmap(..).filter.(..).reduce(..)
+ * Stream[People].groupBy(_.gender).flatMap(..).filter(..).reduce(..)
* }}}
*
- * @param fun Group by function
+ * @param fn Group by function
* @param parallelism Parallelism level
* @param description The description
* @return the grouped stream
*/
- def groupBy[Group](fun: T => Group, parallelism: Int = 1, description:
String = null)
-: Stream[T] = {
-val groupOp = GroupByOp(fun, parallelism,
Option(description).getOrElse("groupBy"))
-graph.addVertex(groupOp)
-graph.addEdge(thisNode, edge.getOrElse(Shuffle), groupOp)
-new Stream[T](graph, groupOp)
+ def groupBy[GROUP](fn: T => GROUP, parallelism: Int = 1, description:
String = "groupBy")
+: GroupByStream[T, GROUP] = {
+groupByWindow(DefaultGroupBy(fn), parallelism, description)
--- End diff --
Yes, I have a new version that does so. Will update today
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #85: [GEARPUMP-23] add window dsl
Github user kkasravi commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/85#discussion_r80745374
--- Diff:
streaming/src/main/scala/org/apache/gearpump/streaming/dsl/Stream.scala ---
@@ -147,6 +150,33 @@ class Stream[T](
graph.addEdge(thisNode, edge.getOrElse(Shuffle), processorOp)
new Stream[R](graph, processorOp, Some(Shuffle))
}
+
+ private def groupByWindow[GROUP](fn: GroupByFn[T, GROUP],
+ parallelism: Int, description: String): GroupByStream[T, GROUP] = {
+val groupOp = GroupByOp[T, GROUP](fn,
+ parallelism, description)
+graph.addVertex(groupOp)
+graph.addEdge(thisNode, edge.getOrElse(Shuffle), groupOp)
+new GroupByStream[T, GROUP](graph, groupOp)
+ }
+
+}
+
+class GroupByStream[T, GROUP](
+private val graph: Graph[Op, OpEdge],
+private val groupByOp: GroupByOp[T, GROUP],
+private val edge: Option[OpEdge] = None) extends Stream[T](graph,
groupByOp, edge) {
--- End diff --
Is 'private val graph' equivalent to 'graph' (val adds public accessors so
private takes them away?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #85: [GEARPUMP-23] add window dsl
Github user kkasravi commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/85#discussion_r80744615
--- Diff:
streaming/src/main/scala/org/apache/gearpump/streaming/dsl/Stream.scala ---
@@ -115,20 +121,17 @@ class Stream[T](
*
* For example,
* {{{
- * Stream[People].groupBy(_.gender).flatmap(..).filter.(..).reduce(..)
+ * Stream[People].groupBy(_.gender).flatMap(..).filter(..).reduce(..)
* }}}
*
- * @param fun Group by function
+ * @param fn Group by function
* @param parallelism Parallelism level
* @param description The description
* @return the grouped stream
*/
- def groupBy[Group](fun: T => Group, parallelism: Int = 1, description:
String = null)
-: Stream[T] = {
-val groupOp = GroupByOp(fun, parallelism,
Option(description).getOrElse("groupBy"))
-graph.addVertex(groupOp)
-graph.addEdge(thisNode, edge.getOrElse(Shuffle), groupOp)
-new Stream[T](graph, groupOp)
+ def groupBy[GROUP](fn: T => GROUP, parallelism: Int = 1, description:
String = "groupBy")
+: GroupByStream[T, GROUP] = {
+groupByWindow(DefaultGroupBy(fn), parallelism, description)
--- End diff --
Seems odd: groupBy -> groupByWindow -> GroupByStream. Suggests that all
groupBy's have windowing semantics.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
