[jira] [Created] (BEAM-1180) Implement GearpumpPipelineResult
Manu Zhang created BEAM-1180: Summary: Implement GearpumpPipelineResult Key: BEAM-1180 URL: https://issues.apache.org/jira/browse/BEAM-1180 Project: Beam Issue Type: Improvement Components: runner-gearpump Reporter: Manu Zhang Assignee: Manu Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1119) Add a trigger that fires when the watermark passes timestamp
[
https://issues.apache.org/jira/browse/BEAM-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manu Zhang updated BEAM-1119:
-
Description:
For a pane with elements and timestamps {{[(a, 1), (b, 4), (c, 5)]}},
* "a" would be emitted if watermark passes 1
* "a" and "b" would be emitted if watermark passes 4
* all the elements would be emitted if watermark passes 5
* if the watermark jumps from 0 to 7 then all the elements would be emitted
* if "d" comes in late with timestamp 3, then it will be emitted if it's within
the allowed lateness.
This usually means the trigger fires in the middle of a window and is useful
when firing at the end of window misses the latency requirement.
was:
For a pane with elements and timestamps {{[(a, 1), (b, 4), (c, 5)]}},
* "a" would be emitted if watermark passes 1
* "a" and "b" would be emitted if watermark passes 4
* all the elements would be emitted if watermark passes 5 or the end of the
window
* if the watermark jumps from 0 to 7 then all the elements would be emitted
* if "d" comes in late with timestamp 3, then it will be emitted if it's within
the allowed lateness.
> Add a trigger that fires when the watermark passes timestamp
>
>
> Key: BEAM-1119
> URL: https://issues.apache.org/jira/browse/BEAM-1119
> Project: Beam
> Issue Type: Improvement
>Reporter: Manu Zhang
>
> For a pane with elements and timestamps {{[(a, 1), (b, 4), (c, 5)]}},
> * "a" would be emitted if watermark passes 1
> * "a" and "b" would be emitted if watermark passes 4
> * all the elements would be emitted if watermark passes 5
> * if the watermark jumps from 0 to 7 then all the elements would be emitted
> * if "d" comes in late with timestamp 3, then it will be emitted if it's
> within the allowed lateness.
> This usually means the trigger fires in the middle of a window and is useful
> when firing at the end of window misses the latency requirement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-1119) Add a trigger that fires when the watermark passes timestamp
[
https://issues.apache.org/jira/browse/BEAM-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manu Zhang updated BEAM-1119:
-
Assignee: Kenneth Knowles
> Add a trigger that fires when the watermark passes timestamp
>
>
> Key: BEAM-1119
> URL: https://issues.apache.org/jira/browse/BEAM-1119
> Project: Beam
> Issue Type: Improvement
>Reporter: Manu Zhang
>Assignee: Kenneth Knowles
>
> For a pane with elements and timestamps {{[(a, 1), (b, 4), (c, 5)]}},
> * "a" would be emitted if watermark passes 1
> * "a" and "b" would be emitted if watermark passes 4
> * all the elements would be emitted if watermark passes 5
> * if the watermark jumps from 0 to 7 then all the elements would be emitted
> * if "d" comes in late with timestamp 3, then it will be emitted if it's
> within the allowed lateness.
> This usually means the trigger fires in the middle of a window and is useful
> when firing at the end of window misses the latency requirement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-1119) Add a trigger that fires when the watermark passes timestamp
[
https://issues.apache.org/jira/browse/BEAM-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manu Zhang updated BEAM-1119:
-
Summary: Add a trigger that fires when the watermark passes timestamp
(was: Add a trigger that fires in the middle of window)
> Add a trigger that fires when the watermark passes timestamp
>
>
> Key: BEAM-1119
> URL: https://issues.apache.org/jira/browse/BEAM-1119
> Project: Beam
> Issue Type: Improvement
>Reporter: Manu Zhang
>
> For a pane with elements and timestamps {{[(a, 1), (b, 4), (c, 5)]}},
> * "a" would be emitted if watermark passes 1
> * "a" and "b" would be emitted if watermark passes 4
> * all the elements would be emitted if watermark passes 5 or the end of the
> window
> * if the watermark jumps from 0 to 7 then all the elements would be emitted
> * if "d" comes in late with timestamp 3, then it will be emitted if it's
> within the allowed lateness.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-1119) Add a trigger that fires in the middle of window
[
https://issues.apache.org/jira/browse/BEAM-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manu Zhang updated BEAM-1119:
-
Description:
For a pane with elements and timestamps {{[(a, 1), (b, 4), (c, 5)]}},
* "a" would be emitted if watermark passes 1
* "a" and "b" would be emitted if watermark passes 4
* all the elements would be emitted if watermark passes 5 or the end of the
window
* if the watermark jumps from 0 to 7 then all the elements would be emitted
* if "d" comes in late with timestamp 3, then it will be emitted if it's within
the allowed lateness.
> Add a trigger that fires in the middle of window
>
>
> Key: BEAM-1119
> URL: https://issues.apache.org/jira/browse/BEAM-1119
> Project: Beam
> Issue Type: Improvement
>Reporter: Manu Zhang
>
> For a pane with elements and timestamps {{[(a, 1), (b, 4), (c, 5)]}},
> * "a" would be emitted if watermark passes 1
> * "a" and "b" would be emitted if watermark passes 4
> * all the elements would be emitted if watermark passes 5 or the end of the
> window
> * if the watermark jumps from 0 to 7 then all the elements would be emitted
> * if "d" comes in late with timestamp 3, then it will be emitted if it's
> within the allowed lateness.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (BEAM-1119) Add a trigger that fires in the middle of window
Manu Zhang created BEAM-1119: Summary: Add a trigger that fires in the middle of window Key: BEAM-1119 URL: https://issues.apache.org/jira/browse/BEAM-1119 Project: Beam Issue Type: Improvement Reporter: Manu Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1086) Upgrade to latest Gearpump snapshot
[ https://issues.apache.org/jira/browse/BEAM-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated BEAM-1086: - Summary: Upgrade to latest Gearpump snapshot (was: Upgrade to latest GEARPUMP snapshots) > Upgrade to latest Gearpump snapshot > --- > > Key: BEAM-1086 > URL: https://issues.apache.org/jira/browse/BEAM-1086 > Project: Beam > Issue Type: Improvement > Components: runner-gearpump >Reporter: Manu Zhang >Assignee: Manu Zhang > > The latest Gearpump snapshot version is available under [apache repo | > https://repository.apache.org/content/repositories/snapshots/org/apache/gearpump/gearpump-core_2.11/0.8.3-SNAPSHOT/]. > To support the Gearpump runner, we need to continuously evolve Gearpump and > bring in Beam capabilities. Depending on a snapshot version will greatly > accelerate the integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1086) Upgrade to latest GEARPUMP snapshots
[ https://issues.apache.org/jira/browse/BEAM-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated BEAM-1086: - Description: The latest Gearpump snapshot version is available under [apache repo | https://repository.apache.org/content/repositories/snapshots/org/apache/gearpump/gearpump-core_2.11/0.8.3-SNAPSHOT/]. To support the Gearpump runner, we need to continuously evolve Gearpump and bring in Beam capabilities. Depending on a snapshot version will greatly accelerate the integration. > Upgrade to latest GEARPUMP snapshots > > > Key: BEAM-1086 > URL: https://issues.apache.org/jira/browse/BEAM-1086 > Project: Beam > Issue Type: Improvement > Components: runner-gearpump >Reporter: Manu Zhang >Assignee: Manu Zhang > > The latest Gearpump snapshot version is available under [apache repo | > https://repository.apache.org/content/repositories/snapshots/org/apache/gearpump/gearpump-core_2.11/0.8.3-SNAPSHOT/]. > To support the Gearpump runner, we need to continuously evolve Gearpump and > bring in Beam capabilities. Depending on a snapshot version will greatly > accelerate the integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1086) Upgrade to latest GEARPUMP snapshots
[ https://issues.apache.org/jira/browse/BEAM-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated BEAM-1086: - Summary: Upgrade to latest GEARPUMP snapshots (was: Upgrade to Gearpump-0.8.2-incubating) > Upgrade to latest GEARPUMP snapshots > > > Key: BEAM-1086 > URL: https://issues.apache.org/jira/browse/BEAM-1086 > Project: Beam > Issue Type: Improvement > Components: runner-gearpump >Reporter: Manu Zhang >Assignee: Manu Zhang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (BEAM-1093) Confusing Javadocs in StateInternals
[ https://issues.apache.org/jira/browse/BEAM-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang reassigned BEAM-1093: Assignee: Manu Zhang (was: Ben Chambers) > Confusing Javadocs in StateInternals > > > Key: BEAM-1093 > URL: https://issues.apache.org/jira/browse/BEAM-1093 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Manu Zhang >Assignee: Manu Zhang >Priority: Minor > > At last but one line of StateInternals' Javadocs, it says "This is a > low-level API intended for use by the Dataflow SDK". Not sure what is > "Dataflow SDK". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1093) Confusing Javadocs in StateInternals
[ https://issues.apache.org/jira/browse/BEAM-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated BEAM-1093: - Assignee: Ben Chambers (was: Davor Bonaci) > Confusing Javadocs in StateInternals > > > Key: BEAM-1093 > URL: https://issues.apache.org/jira/browse/BEAM-1093 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Manu Zhang >Assignee: Ben Chambers >Priority: Minor > > At last but one line of StateInternals' Javadocs, it says "This is a > low-level API intended for use by the Dataflow SDK". Not sure what is > "Dataflow SDK". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1093) Confusing Javadocs in StateInternals
Manu Zhang created BEAM-1093: Summary: Confusing Javadocs in StateInternals Key: BEAM-1093 URL: https://issues.apache.org/jira/browse/BEAM-1093 Project: Beam Issue Type: Improvement Components: sdk-java-core Reporter: Manu Zhang Assignee: Davor Bonaci Priority: Minor At last but one line of StateInternals' Javadocs, it says "This is a low-level API intended for use by the Dataflow SDK". Not sure what is "Dataflow SDK". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1086) Upgrade to Gearpump-0.8.2-incubating
Manu Zhang created BEAM-1086: Summary: Upgrade to Gearpump-0.8.2-incubating Key: BEAM-1086 URL: https://issues.apache.org/jira/browse/BEAM-1086 Project: Beam Issue Type: Improvement Components: runner-gearpump Reporter: Manu Zhang Assignee: Manu Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-9) Storm Runner
[ https://issues.apache.org/jira/browse/BEAM-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721082#comment-15721082 ] Manu Zhang commented on BEAM-9: --- Here is a [feature-branch|https://github.com/apache/storm/tree/beam-runner/external/storm-beam] on Storm that you can watch on. AFAIK, the storm community prefer to develop and put the runner at Storm's side. > Storm Runner > > > Key: BEAM-9 > URL: https://issues.apache.org/jira/browse/BEAM-9 > Project: Beam > Issue Type: Wish > Components: runner-ideas >Reporter: Frances Perry >Assignee: Sriharsha Chintalapani > > Gathering place for interest in a Storm runner for Beam. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1043) Dependency problems found in runner-apex
Manu Zhang created BEAM-1043: Summary: Dependency problems found in runner-apex Key: BEAM-1043 URL: https://issues.apache.org/jira/browse/BEAM-1043 Project: Beam Issue Type: Bug Components: runner-apex Affects Versions: 0.3.0-incubating Reporter: Manu Zhang Assignee: Thomas Weise [INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ beam-runners-apex --- [WARNING] Used undeclared dependencies found: [WARNING]commons-io:commons-io:jar:2.4:compile [WARNING]com.datatorrent:netlet:jar:1.3.0:compile [WARNING]org.apache.hadoop:hadoop-common:jar:2.6.0:compile -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-973) Add end user and developer documentation to gearpump-runner
Manu Zhang created BEAM-973: --- Summary: Add end user and developer documentation to gearpump-runner Key: BEAM-973 URL: https://issues.apache.org/jira/browse/BEAM-973 Project: Beam Issue Type: Sub-task Components: runner-gearpump Affects Versions: 0.4.0-incubating Reporter: Manu Zhang Assignee: Manu Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-972) Add basic level of unit testing to gearpump runner
Manu Zhang created BEAM-972: --- Summary: Add basic level of unit testing to gearpump runner Key: BEAM-972 URL: https://issues.apache.org/jira/browse/BEAM-972 Project: Beam Issue Type: Sub-task Components: runner-gearpump, testing Affects Versions: 0.4.0-incubating Reporter: Manu Zhang Assignee: Manu Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-971) Add support for merging windows in gearpump runner
Manu Zhang created BEAM-971: --- Summary: Add support for merging windows in gearpump runner Key: BEAM-971 URL: https://issues.apache.org/jira/browse/BEAM-971 Project: Beam Issue Type: Sub-task Components: runner-gearpump Affects Versions: 0.4.0-incubating Reporter: Manu Zhang Assignee: Manu Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-970) Add side-input/side-output support to gearpump-runner
Manu Zhang created BEAM-970: --- Summary: Add side-input/side-output support to gearpump-runner Key: BEAM-970 URL: https://issues.apache.org/jira/browse/BEAM-970 Project: Beam Issue Type: Sub-task Components: runner-gearpump Affects Versions: 0.4.0-incubating Reporter: Manu Zhang Assignee: Manu Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-969) Add a gearpump runner web page under "learn/runners"
Manu Zhang created BEAM-969: --- Summary: Add a gearpump runner web page under "learn/runners" Key: BEAM-969 URL: https://issues.apache.org/jira/browse/BEAM-969 Project: Beam Issue Type: Sub-task Components: runner-gearpump, website Reporter: Manu Zhang Assignee: Manu Zhang Fix For: 0.4.0-incubating -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-968) Update capability matrix to include gearpump-runner
[ https://issues.apache.org/jira/browse/BEAM-968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated BEAM-968: Issue Type: Sub-task (was: Bug) Parent: BEAM-79 > Update capability matrix to include gearpump-runner > --- > > Key: BEAM-968 > URL: https://issues.apache.org/jira/browse/BEAM-968 > Project: Beam > Issue Type: Sub-task > Components: runner-gearpump, website >Affects Versions: 0.4.0-incubating >Reporter: Manu Zhang >Assignee: Manu Zhang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-968) Update capability matrix to include gearpump-runner
Manu Zhang created BEAM-968: --- Summary: Update capability matrix to include gearpump-runner Key: BEAM-968 URL: https://issues.apache.org/jira/browse/BEAM-968 Project: Beam Issue Type: Bug Components: runner-gearpump, website Affects Versions: 0.4.0-incubating Reporter: Manu Zhang Assignee: Manu Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-942) Gearpump test configuration should exclude merging window tests
[
https://issues.apache.org/jira/browse/BEAM-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15646474#comment-15646474
]
Manu Zhang commented on BEAM-942:
-
I guess [these
lines|https://github.com/apache/incubator-beam/blob/gearpump-runner/runners/gearpump/pom.xml#L74]
to exclude tests no longer works ?
> Gearpump test configuration should exclude merging window tests
> ---
>
> Key: BEAM-942
> URL: https://issues.apache.org/jira/browse/BEAM-942
> Project: Beam
> Issue Type: Bug
> Components: runner-gearpump
>Reporter: Kenneth Knowles
>Assignee: Manu Zhang
>
> Looking at [this Jenkins
> log|https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GearpumpLocal/152/consoleFull]
> we can see various messages about unsupported features. We should exclude
> these tests so that the Jenkins results are more helpful.
> {code}
> java.lang.UnsupportedOperationException: merging window is not supported in
> Gearpump pipeline
> {code}
> and
> {code}
> java.lang.IllegalStateException: no translator registered for
> View.CreatePCollectionView
> {code}
> These are clearly "side inputs" and "merging windows". And I think that would
> cover all the failures.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-800) DoFnLifeCycleManager should hold DoFnInvoker, not DoFn/OldDoFn
[
https://issues.apache.org/jira/browse/BEAM-800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15617139#comment-15617139
]
Manu Zhang commented on BEAM-800:
-
Ok. The remaining issues are that {{ParDoEvaluator}} accepts {{DoFn}} now
(BEAM-788 ?) and {{DoFnInvoker}} is not {{Serializable}}
> DoFnLifeCycleManager should hold DoFnInvoker, not DoFn/OldDoFn
> --
>
> Key: BEAM-800
> URL: https://issues.apache.org/jira/browse/BEAM-800
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Manu Zhang
>Priority: Minor
>
> The {{DirectRunner}}'s support class {{DoFnLifecycleManager}} holds a cache
> of deserialized {{OldDoFn}} s, now being ported to {{DoFn}} s. But to execute
> a {{DoFn}} there is another layer of indirection through a {{DoFnInvoker}},
> which is the best object to cache here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-800) DoFnLifeCycleManager should hold DoFnInvoker, not DoFn/OldDoFn
[
https://issues.apache.org/jira/browse/BEAM-800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15617138#comment-15617138
]
Manu Zhang commented on BEAM-800:
-
Ok. The remaining issues are that {{ParDoEvaluator}} accepts {{DoFn}} now
(BEAM-788 ?) and {{DoFnInvoker}} is not {{Serializable}}
> DoFnLifeCycleManager should hold DoFnInvoker, not DoFn/OldDoFn
> --
>
> Key: BEAM-800
> URL: https://issues.apache.org/jira/browse/BEAM-800
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Manu Zhang
>Priority: Minor
>
> The {{DirectRunner}}'s support class {{DoFnLifecycleManager}} holds a cache
> of deserialized {{OldDoFn}} s, now being ported to {{DoFn}} s. But to execute
> a {{DoFn}} there is another layer of indirection through a {{DoFnInvoker}},
> which is the best object to cache here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (BEAM-800) DoFnLifeCycleManager should hold DoFnInvoker, not DoFn/OldDoFn
[
https://issues.apache.org/jira/browse/BEAM-800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manu Zhang reassigned BEAM-800:
---
Assignee: Manu Zhang
> DoFnLifeCycleManager should hold DoFnInvoker, not DoFn/OldDoFn
> --
>
> Key: BEAM-800
> URL: https://issues.apache.org/jira/browse/BEAM-800
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Manu Zhang
>Priority: Minor
>
> The {{DirectRunner}}'s support class {{DoFnLifecycleManager}} holds a cache
> of deserialized {{OldDoFn}} s, now being ported to {{DoFn}} s. But to execute
> a {{DoFn}} there is another layer of indirection through a {{DoFnInvoker}},
> which is the best object to cache here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-800) DoFnLifeCycleManager should hold DoFnInvoker, not DoFn/OldDoFn
[
https://issues.apache.org/jira/browse/BEAM-800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15613941#comment-15613941
]
Manu Zhang commented on BEAM-800:
-
{{DoFnRunner}} is currently created by the corresponding
{{TransformEvaluator}}. Do you want to move it to {{TransformEvaluatorFactory}}
?
> DoFnLifeCycleManager should hold DoFnInvoker, not DoFn/OldDoFn
> --
>
> Key: BEAM-800
> URL: https://issues.apache.org/jira/browse/BEAM-800
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
>Reporter: Kenneth Knowles
>Priority: Minor
>
> The {{DirectRunner}}'s support class {{DoFnLifecycleManager}} holds a cache
> of deserialized {{OldDoFn}} s, now being ported to {{DoFn}} s. But to execute
> a {{DoFn}} there is another layer of indirection through a {{DoFnInvoker}},
> which is the best object to cache here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (BEAM-836) JavaDocs for DoFnRunner is not updated
Manu Zhang created BEAM-836:
---
Summary: JavaDocs for DoFnRunner is not updated
Key: BEAM-836
URL: https://issues.apache.org/jira/browse/BEAM-836
Project: Beam
Issue Type: Bug
Components: runner-core
Affects Versions: 0.3.0-incubating
Reporter: Manu Zhang
Assignee: Frances Perry
Priority: Minor
It's still for the {{OldDoFn}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-800) DoFnLifeCycleManager should hold DoFnInvoker, not DoFn/OldDoFn
[
https://issues.apache.org/jira/browse/BEAM-800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601003#comment-15601003
]
Manu Zhang commented on BEAM-800:
-
Could you link where {{OldDoFn}} being ported to {{DoFn}} happens ? Is this
blocked by the porting ?
> DoFnLifeCycleManager should hold DoFnInvoker, not DoFn/OldDoFn
> --
>
> Key: BEAM-800
> URL: https://issues.apache.org/jira/browse/BEAM-800
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
>Reporter: Kenneth Knowles
>Priority: Minor
>
> The {{DirectRunner}}'s support class {{DoFnLifecycleManager}} holds a cache
> of deserialized {{OldDoFn}} s, now being ported to {{DoFn}} s. But to execute
> a {{DoFn}} there is another layer of indirection through a {{DoFnInvoker}},
> which is the best object to cache here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-800) DoFnLifeCycleManager should hold DoFnInvoker, not DoFn/OldDoFn
[
https://issues.apache.org/jira/browse/BEAM-800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manu Zhang updated BEAM-800:
Description: The {{DirectRunner}}'s support class {{DoFnLifecycleManager}}
holds a cache of deserialized {{OldDoFn}} s, now being ported to {{DoFn}} s.
But to execute a {{DoFn}} there is another layer of indirection through a
{{DoFnInvoker}}, which is the best object to cache here. (was: The
{{DirectRunner}}'s support class {{DoFnLifecycleManager}} holds a cache of
deserialized {{OldDoFn}}s, now being ported to {{DoFn}}s. But to execute a
{{DoFn}} there is another layer of indirection through a {{DoFnInvoker}}, which
is the best object to cache here.)
> DoFnLifeCycleManager should hold DoFnInvoker, not DoFn/OldDoFn
> --
>
> Key: BEAM-800
> URL: https://issues.apache.org/jira/browse/BEAM-800
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
>Reporter: Kenneth Knowles
>Priority: Minor
>
> The {{DirectRunner}}'s support class {{DoFnLifecycleManager}} holds a cache
> of deserialized {{OldDoFn}} s, now being ported to {{DoFn}} s. But to execute
> a {{DoFn}} there is another layer of indirection through a {{DoFnInvoker}},
> which is the best object to cache here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (BEAM-740) Improve document on running Beam program on Flink Cluster
Manu Zhang created BEAM-740:
---
Summary: Improve document on running Beam program on Flink Cluster
Key: BEAM-740
URL: https://issues.apache.org/jira/browse/BEAM-740
Project: Beam
Issue Type: Improvement
Components: runner-flink
Affects Versions: 0.2.0-incubating
Reporter: Manu Zhang
Priority: Trivial
The uber jar to run on Flink cluster contains a {{LICENSE}} file and a
{{license}} directory. The two will conflict on the case-insensitive file
system of OS X when the jar file is being extracted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (BEAM-631) upgrade surefire plugin to 2.19.1
Manu Zhang created BEAM-631: --- Summary: upgrade surefire plugin to 2.19.1 Key: BEAM-631 URL: https://issues.apache.org/jira/browse/BEAM-631 Project: Beam Issue Type: Improvement Components: testing Affects Versions: 0.2.0-incubating Reporter: Manu Zhang Assignee: Manu Zhang Priority: Minor As of surefire 2.19.1, fully qualified class names and packages can be used to exclude tests, which is needed by Gearpump Runner. Spark Runner also requires 2.19.1 for fork control. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-440) Create.values() returns a type-unsafe Coder
[
https://issues.apache.org/jira/browse/BEAM-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manu Zhang updated BEAM-440:
Assignee: (was: Manu Zhang)
> Create.values() returns a type-unsafe Coder
> ---
>
> Key: BEAM-440
> URL: https://issues.apache.org/jira/browse/BEAM-440
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Daniel Halperin
> Labels: newbie, starter
>
> Create.values() with no arguments will default to a VoidCoder, unless one is
> set later with #setCoder(Coder).
> Although it will encode its input correctly, this seems like a bad choice in
> many cases. E.g., with Flatten:
> PCollection> initial = p.apply("First",
> Create.>of());
> PCollection> second =
> p.apply("Second", Create.of("a", "b")).apply(ParDo.of(new
> MyAvroDoFn()));
> PCollectionList
> .of(initial).and(second)
> .apply(Flatten.>pCollections());
> This crashes trying to cast a KV from "Second" to a Void.class.
> 1. Suggest throwing a warning in #getDefaultOutputCoder when defaulting to
> VoidCoder for an empty elements list. Should this be an error?
> 2. Suggest adding something like Create.empty(TypeDescriptor) to handle this
> case properly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (BEAM-440) Create.values() returns a type-unsafe Coder
[
https://issues.apache.org/jira/browse/BEAM-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manu Zhang reassigned BEAM-440:
---
Assignee: Manu Zhang
> Create.values() returns a type-unsafe Coder
> ---
>
> Key: BEAM-440
> URL: https://issues.apache.org/jira/browse/BEAM-440
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Manu Zhang
> Labels: newbie, starter
>
> Create.values() with no arguments will default to a VoidCoder, unless one is
> set later with #setCoder(Coder).
> Although it will encode its input correctly, this seems like a bad choice in
> many cases. E.g., with Flatten:
> PCollection> initial = p.apply("First",
> Create.>of());
> PCollection> second =
> p.apply("Second", Create.of("a", "b")).apply(ParDo.of(new
> MyAvroDoFn()));
> PCollectionList
> .of(initial).and(second)
> .apply(Flatten.>pCollections());
> This crashes trying to cast a KV from "Second" to a Void.class.
> 1. Suggest throwing a warning in #getDefaultOutputCoder when defaulting to
> VoidCoder for an empty elements list. Should this be an error?
> 2. Suggest adding something like Create.empty(TypeDescriptor) to handle this
> case properly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (BEAM-489) remove headerLocation from maven-checkstyle-plugin configuration
Manu Zhang created BEAM-489: --- Summary: remove headerLocation from maven-checkstyle-plugin configuration Key: BEAM-489 URL: https://issues.apache.org/jira/browse/BEAM-489 Project: Beam Issue Type: Improvement Components: build-system Affects Versions: 0.1.0-incubating Reporter: Manu Zhang Assignee: Manu Zhang Priority: Trivial As license header has been checked by maven-rat-plugin since BEAM-254, headerLocation in the maven-checkstyle-plugin is redundant. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (BEAM-450) Modules are shaded to the same path
[
https://issues.apache.org/jira/browse/BEAM-450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manu Zhang reassigned BEAM-450:
---
Assignee: Manu Zhang
> Modules are shaded to the same path
> ---
>
> Key: BEAM-450
> URL: https://issues.apache.org/jira/browse/BEAM-450
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating
>Reporter: Daniel Halperin
>Assignee: Manu Zhang
> Labels: newbie, starter
>
> Right now multiple modules are using the same repackaged path. We should be
> using per-artifact paths so that they don't conflict.
> One proposal was simply to adopt
> {{${project.groupId}.${project.artifactId}.repackaged}} as the shading
> location. If it works.
> This is a good starter issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-328) CoderRegistry does not provide SerializableCoder for `T extends Serializable`
[
https://issues.apache.org/jira/browse/BEAM-328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15333697#comment-15333697
]
Manu Zhang commented on BEAM-328:
-
It seems that {{TypeVariable}} cannot be cast a {{Class}}
> CoderRegistry does not provide SerializableCoder for `T extends Serializable`
> -
>
> Key: BEAM-328
> URL: https://issues.apache.org/jira/browse/BEAM-328
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Manu Zhang
>Priority: Minor
> Labels: starter
>
> When the type for which a coder is being inferred is a type variable with an
> upper bound of {{Serializable}}, it is reasonable for the coder registry to
> propagate this to the {{SerializableCoder.PROVIDER}}, which should be able to
> succeed.
> Unfortunately, the particulars of the distinctions made between {{Type}},
> {{Class}}, {{TypeVariable}}, {{ParameterizedType}}, etc, go down a code path
> where this is not the case. Instead, an error is raised that the type
> variable has been subject to erasure.
> Originally reported at:
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/298
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (BEAM-328) CoderRegistry does not provide SerializableCoder for `T extends Serializable`
[
https://issues.apache.org/jira/browse/BEAM-328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15333697#comment-15333697
]
Manu Zhang edited comment on BEAM-328 at 6/16/16 12:40 PM:
---
It seems that {{TypeVariable}} cannot be cast to a {{Class}}
was (Author: mauzhang):
It seems that {{TypeVariable}} cannot be cast a {{Class}}
> CoderRegistry does not provide SerializableCoder for `T extends Serializable`
> -
>
> Key: BEAM-328
> URL: https://issues.apache.org/jira/browse/BEAM-328
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Manu Zhang
>Priority: Minor
> Labels: starter
>
> When the type for which a coder is being inferred is a type variable with an
> upper bound of {{Serializable}}, it is reasonable for the coder registry to
> propagate this to the {{SerializableCoder.PROVIDER}}, which should be able to
> succeed.
> Unfortunately, the particulars of the distinctions made between {{Type}},
> {{Class}}, {{TypeVariable}}, {{ParameterizedType}}, etc, go down a code path
> where this is not the case. Instead, an error is raised that the type
> variable has been subject to erasure.
> Originally reported at:
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/298
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (BEAM-328) CoderRegistry does not provide SerializableCoder for `T extends Serializable`
[
https://issues.apache.org/jira/browse/BEAM-328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manu Zhang reassigned BEAM-328:
---
Assignee: Manu Zhang
> CoderRegistry does not provide SerializableCoder for `T extends Serializable`
> -
>
> Key: BEAM-328
> URL: https://issues.apache.org/jira/browse/BEAM-328
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Manu Zhang
>Priority: Minor
> Labels: starter
>
> When the type for which a coder is being inferred is a type variable with an
> upper bound of {{Serializable}}, it is reasonable for the coder registry to
> propagate this to the {{SerializableCoder.PROVIDER}}, which should be able to
> succeed.
> Unfortunately, the particulars of the distinctions made between {{Type}},
> {{Class}}, {{TypeVariable}}, {{ParameterizedType}}, etc, go down a code path
> where this is not the case. Instead, an error is raised that the type
> variable has been subject to erasure.
> Originally reported at:
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/298
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-332) Matcher(s) for TableRow
[
https://issues.apache.org/jira/browse/BEAM-332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331516#comment-15331516
]
Manu Zhang commented on BEAM-332:
-
Why not have a good {{equals}} in {{TableRow}} itself ?
> Matcher(s) for TableRow
> ---
>
> Key: BEAM-332
> URL: https://issues.apache.org/jira/browse/BEAM-332
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Minor
> Labels: starter
>
> {{TableRow}} has poorly behaved equality, and a certain amount of automatic
> coercion, so matchers based equals() are not applicable in tests. It would be
> handy to have matchers such as "isTableRowEqualTo(otherRow)".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-342) Change Filter#greaterThan, etc. to actually use Filter
[
https://issues.apache.org/jira/browse/BEAM-342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331274#comment-15331274
]
Manu Zhang commented on BEAM-342:
-
I'd like to take this starter task and I've assigned it to myself without
asking for permissions. Is that fine ?
> Change Filter#greaterThan, etc. to actually use Filter
> --
>
> Key: BEAM-342
> URL: https://issues.apache.org/jira/browse/BEAM-342
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Manu Zhang
>Priority: Minor
> Labels: starter
>
> This is a good starter task.
> Right now,
> [{{Filter#greaterThan}}|https://github.com/apache/incubator-beam/blob/315b3c8e333e5f42730c19e89f856d778ce93cab/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Filter.java#L134]
> constructs a new DoFn rather than using {{Filter#byPredicate}}. We should
> fix this to make it consistent and simpler.
> We can also remove deprecated functions in that file, and if possible
> redundant display data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (BEAM-342) Change Filter#greaterThan, etc. to actually use Filter
[
https://issues.apache.org/jira/browse/BEAM-342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manu Zhang reassigned BEAM-342:
---
Assignee: Manu Zhang
> Change Filter#greaterThan, etc. to actually use Filter
> --
>
> Key: BEAM-342
> URL: https://issues.apache.org/jira/browse/BEAM-342
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Manu Zhang
>Priority: Minor
> Labels: starter
>
> This is a good starter task.
> Right now,
> [{{Filter#greaterThan}}|https://github.com/apache/incubator-beam/blob/315b3c8e333e5f42730c19e89f856d778ce93cab/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Filter.java#L134]
> constructs a new DoFn rather than using {{Filter#byPredicate}}. We should
> fix this to make it consistent and simpler.
> We can also remove deprecated functions in that file, and if possible
> redundant display data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-336) examples-java README is not up-to-date
[ https://issues.apache.org/jira/browse/BEAM-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated BEAM-336: Fix Version/s: 0.2.0-incubating > examples-java README is not up-to-date > -- > > Key: BEAM-336 > URL: https://issues.apache.org/jira/browse/BEAM-336 > Project: Beam > Issue Type: Bug > Components: examples-java >Affects Versions: 0.1.0-incubating >Reporter: Manu Zhang >Assignee: Manu Zhang >Priority: Minor > Fix For: 0.2.0-incubating > > > I found some issues when following the examples-java README > * class name is the old "com.google.cloud.dataflow.examples.WordCount" > * "mvn compile exec:java -pl examples" failed with "ClassNotFoundException: > org.apache.beam.examples.WordCount" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-336) examples-java README is not up-to-date
[ https://issues.apache.org/jira/browse/BEAM-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang resolved BEAM-336. - Resolution: Fixed > examples-java README is not up-to-date > -- > > Key: BEAM-336 > URL: https://issues.apache.org/jira/browse/BEAM-336 > Project: Beam > Issue Type: Bug > Components: examples-java >Affects Versions: 0.1.0-incubating >Reporter: Manu Zhang >Assignee: Manu Zhang >Priority: Minor > > I found some issues when following the examples-java README > * class name is the old "com.google.cloud.dataflow.examples.WordCount" > * "mvn compile exec:java -pl examples" failed with "ClassNotFoundException: > org.apache.beam.examples.WordCount" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-336) examples-java README is not up-to-date
[ https://issues.apache.org/jira/browse/BEAM-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated BEAM-336: Description: I found some issues when following the examples-java README * class name is the old "com.google.cloud.dataflow.examples.WordCount" * "mvn compile exec:java -pl examples" failed with "ClassNotFoundException: org.apache.beam.examples.WordCount" > examples-java README is not up-to-date > -- > > Key: BEAM-336 > URL: https://issues.apache.org/jira/browse/BEAM-336 > Project: Beam > Issue Type: Bug > Components: examples-java >Affects Versions: 0.1.0-incubating >Reporter: Manu Zhang >Assignee: Manu Zhang >Priority: Minor > > I found some issues when following the examples-java README > * class name is the old "com.google.cloud.dataflow.examples.WordCount" > * "mvn compile exec:java -pl examples" failed with "ClassNotFoundException: > org.apache.beam.examples.WordCount" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-336) examples-java README is not up-to-date
Manu Zhang created BEAM-336: --- Summary: examples-java README is not up-to-date Key: BEAM-336 URL: https://issues.apache.org/jira/browse/BEAM-336 Project: Beam Issue Type: Bug Components: examples-java Affects Versions: 0.1.0-incubating Reporter: Manu Zhang Assignee: Manu Zhang Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-79) Gearpump runner
[ https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229667#comment-15229667 ] Manu Zhang commented on BEAM-79: Here is a [design document | https://drive.google.com/open?id=1nw64QUWVfT8L7FUprPGLEeNjSBpDMkn1otfLt2rHM5g]. Please kindly comment it. > Gearpump runner > --- > > Key: BEAM-79 > URL: https://issues.apache.org/jira/browse/BEAM-79 > Project: Beam > Issue Type: New Feature > Components: runner-ideas >Reporter: Tyler Akidau >Assignee: Manu Zhang > > Intel is submitting Gearpump (http://www.gearpump.io) to ASF > (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of > low-level primitives a la MillWheel, with some higher level primitives like > non-merging windowing mixed in. Seems like it would make a nice Beam runner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-79) Gearpump runner
[ https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15173266#comment-15173266 ] Manu Zhang commented on BEAM-79: Thanks. I'm glad to take this task, and the Gearpump team is also very excited. I'll follow the guides and reach out for you guys whenever in doubts. > Gearpump runner > --- > > Key: BEAM-79 > URL: https://issues.apache.org/jira/browse/BEAM-79 > Project: Beam > Issue Type: New Feature > Components: runner-ideas >Reporter: Tyler Akidau >Assignee: Manu Zhang > > Intel is submitting Gearpump (http://www.gearpump.io) to ASF > (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of > low-level primitives a la MillWheel, with some higher level primitives like > non-merging windowing mixed in. Seems like it would make a nice Beam runner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-79) Gearpump runner
[ https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170502#comment-15170502 ] Manu Zhang commented on BEAM-79: This is manuzhang from Gearpump team. May I take this task ? > Gearpump runner > --- > > Key: BEAM-79 > URL: https://issues.apache.org/jira/browse/BEAM-79 > Project: Beam > Issue Type: New Feature > Components: runner-ideas >Reporter: Tyler Akidau > > Intel is submitting Gearpump (http://www.gearpump.io) to ASF > (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of > low-level primitives a la MillWheel, with some higher level primitives like > non-merging windowing mixed in. Seems like it would make a nice Beam runner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
