[jira] [Assigned] (BEAM-1198) ViewFn: explicitly decouple runner materialization of side inputs from SDK-specific mapping
[
https://issues.apache.org/jira/browse/BEAM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-1198:
-
Assignee: Kenneth Knowles
> ViewFn: explicitly decouple runner materialization of side inputs from
> SDK-specific mapping
> ---
>
> Key: BEAM-1198
> URL: https://issues.apache.org/jira/browse/BEAM-1198
> Project: Beam
> Issue Type: New Feature
> Components: beam-model-fn-api, beam-model-runner-api
> Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> For side inputs, the field {{PCollectionView.fromIterableInternal}} implies
> an "iterable" materialization of the contents of a PCollection, which is
> adapted to the desired user-facing type within a UDF (today the
> PCollectionView "is" the UDF)
> In practice, runners get adequate performance by special casing just a couple
> of types of PCollectionView in a rather cumbersome manner.
> The design to improve this is https://s.apache.org/beam-side-inputs-1-pager
> and this makes the de facto standard approach the actual model.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (BEAM-846) Decouple side input window mapping from WindowFn
[ https://issues.apache.org/jira/browse/BEAM-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15613384#comment-15613384 ] Kenneth Knowles edited comment on BEAM-846 at 12/21/16 9:19 PM: Design 1-pager is https://s.apache.org/beam-windowmappingfn-1-pager was (Author: kenn): Design 1-pager is https://s.apache.org/beam-side-inputs-1-pager and a couple PRs have been authored ([#520|https://github.com/apache/incubator-beam/pull/520] and [#1076|https://github.com/apache/incubator-beam/pull/1076]) attributed to BEAM-115 (the "all of the Runner API" ticket) > Decouple side input window mapping from WindowFn > > > Key: BEAM-846 > URL: https://issues.apache.org/jira/browse/BEAM-846 > Project: Beam > Issue Type: New Feature > Components: beam-model-runner-api, sdk-java-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles > Labels: backward-incompatible > > Currently the main WindowFn provides as getSideInputWindow method. Instead, > this mapping should be specified per-side-input (thought the default mapping > would remain the same). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-210) Allow control of empty ON_TIME panes analogous to final panes
[ https://issues.apache.org/jira/browse/BEAM-210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-210: - Component/s: (was: beam-model) > Allow control of empty ON_TIME panes analogous to final panes > - > > Key: BEAM-210 > URL: https://issues.apache.org/jira/browse/BEAM-210 > Project: Beam > Issue Type: Bug > Components: beam-model-runner-api, sdk-java-core >Reporter: Mark Shields >Assignee: Thomas Groh > > Today, ON_TIME panes are emitted whether or not they are empty. We had > decided that for final panes the user would want to control this behavior, to > control data volume. But for ON_TIME panes no such control exists. The > rationale is perhaps that the ON_TIME pane is a fundamental result that > should not be elided. To be considered: whether this is what we want. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-260) WindowMappingFn: Know the getSideInputWindow upper bound to release side input resources
[ https://issues.apache.org/jira/browse/BEAM-260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-260: - Component/s: (was: beam-model) beam-model-fn-api > WindowMappingFn: Know the getSideInputWindow upper bound to release side > input resources > > > Key: BEAM-260 > URL: https://issues.apache.org/jira/browse/BEAM-260 > Project: Beam > Issue Type: Bug > Components: beam-model-fn-api, beam-model-runner-api >Reporter: Mark Shields >Assignee: Kenneth Knowles > > We currently have no static knowledge about the getSideInputWindow function, > and runners are thus forced to hold on to all side input state / elements in > case a future element reaches back into an earlier side input element. > Maybe we need an upper bound on lag from current to result of > getSideInputWindow so we can have a progressing gc horizon as we do for GKB > window state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-846) Decouple side input window mapping from WindowFn
[ https://issues.apache.org/jira/browse/BEAM-846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-846: - Component/s: (was: beam-model) > Decouple side input window mapping from WindowFn > > > Key: BEAM-846 > URL: https://issues.apache.org/jira/browse/BEAM-846 > Project: Beam > Issue Type: New Feature > Components: beam-model-runner-api, sdk-java-core >Reporter: Robert Bradshaw > Assignee: Kenneth Knowles > Labels: backward-incompatible > > Currently the main WindowFn provides as getSideInputWindow method. Instead, > this mapping should be specified per-side-input (thought the default mapping > would remain the same). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1193) Give Coders URNs and document their binary formats outside the Java code base
[ https://issues.apache.org/jira/browse/BEAM-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-1193: -- Component/s: (was: beam-model) > Give Coders URNs and document their binary formats outside the Java code base > - > > Key: BEAM-1193 > URL: https://issues.apache.org/jira/browse/BEAM-1193 > Project: Beam > Issue Type: New Feature > Components: beam-model-runner-api > Reporter: Kenneth Knowles >Assignee: Frances Perry > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-653) Refine specification for WindowFn.isCompatible()
[
https://issues.apache.org/jira/browse/BEAM-653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-653:
-
Component/s: beam-model-runner-api
> Refine specification for WindowFn.isCompatible()
> -
>
> Key: BEAM-653
> URL: https://issues.apache.org/jira/browse/BEAM-653
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, beam-model-runner-api
> Reporter: Kenneth Knowles
>
> {{WindowFn#isCompatible}} doesn't really have a spec. In practice, it is used
> primarily when flattening together multiple PCollections. All of the
> WindowFns must be compatible, and then just a single WindowFn is selected
> arbitrarily for the output PCollection.
> In consequence, downstream of the Flatten, the merging behavior will be taken
> from this WindowFn.
> Currently, there are some mismatches:
> - Sessions with different gap durations _are_ compatible today, but probably
> shouldn't be since merging makes little sense. (The use of tiny proto-windows
> is an implementation detail anyhow)
> - SlidingWindows and FixedWindows _could_ reasonably be compatible if they
> had the same duration, though it might be odd.
> Either way, we should just nail down what we actually mean so we can arrive
> at a verdict in these cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-1149) Side input access fails in direct runner (possibly others too) when input element in multiple windows
[
https://issues.apache.org/jira/browse/BEAM-1149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-1149:
--
Component/s: runner-core
> Side input access fails in direct runner (possibly others too) when input
> element in multiple windows
> -
>
> Key: BEAM-1149
> URL: https://issues.apache.org/jira/browse/BEAM-1149
> Project: Beam
> Issue Type: Bug
> Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 0.4.0-incubating
>
>
> {code:java}
> private static class FnWithSideInputs extends DoFn<String, String> {
> private final PCollectionView view;
> private FnWithSideInputs(PCollectionView view) {
> this.view = view;
> }
> @ProcessElement
> public void processElement(ProcessContext c) {
> c.output(c.element() + ":" + c.sideInput(view));
> }
> }
> @Test
> public void testSideInputsWithMultipleWindows() {
> Pipeline p = TestPipeline.create();
> MutableDateTime mutableNow = Instant.now().toMutableDateTime();
> mutableNow.setMillisOfSecond(0);
> Instant now = mutableNow.toInstant();
> SlidingWindows windowFn =
>
> SlidingWindows.of(Duration.standardSeconds(5)).every(Duration.standardSeconds(1));
> PCollectionView view =
> p.apply(Create.of(1)).apply(View.asSingleton());
> PCollection res =
> p.apply(Create.timestamped(TimestampedValue.of("a", now)))
> .apply(Window.into(windowFn))
> .apply(ParDo.of(new FnWithSideInputs(view)).withSideInputs(view));
> PAssert.that(res).containsInAnyOrder("a:1");
> p.run();
> }
> {code}
> This fails with the following exception:
> {code}
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.IllegalStateException: sideInput called when main input element is
> in multiple windows
> at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:343)
> at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:1)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:176)
> at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:112)
> at
> Caused by: java.lang.IllegalStateException: sideInput called when main input
> element is in multiple windows
> at
> org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.sideInput(SimpleDoFnRunner.java:514)
> at
> org.apache.beam.sdk.transforms.ParDoTest$FnWithSideInputs.processElement(ParDoTest.java:738)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (BEAM-1198) ViewFn: explicitly decouple runner materialization of side inputs from SDK-specific mapping
Kenneth Knowles created BEAM-1198:
-
Summary: ViewFn: explicitly decouple runner materialization of
side inputs from SDK-specific mapping
Key: BEAM-1198
URL: https://issues.apache.org/jira/browse/BEAM-1198
Project: Beam
Issue Type: New Feature
Components: beam-model-fn-api, beam-model-runner-api
Reporter: Kenneth Knowles
For side inputs, the field {{PCollectionView.fromIterableInternal}} implies an
"iterable" materialization of the contents of a PCollection, which is adapted
to the desired user-facing type within a UDF (today the PCollectionView "is"
the UDF)
In practice, runners get adequate performance by special casing just a couple
of types of PCollectionView in a rather cumbersome manner.
The design to improve this is https://s.apache.org/beam-side-inputs-1-pager and
this makes the de facto standard approach the actual model.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (BEAM-1003) Enable caching of side-input dependent computations
[ https://issues.apache.org/jira/browse/BEAM-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles closed BEAM-1003. - Resolution: Duplicate Fix Version/s: Not applicable Looks like an exact duplicate, a la form resubmission. > Enable caching of side-input dependent computations > --- > > Key: BEAM-1003 > URL: https://issues.apache.org/jira/browse/BEAM-1003 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Frances Perry > Fix For: Not applicable > > > Sometimes the kind of computations one wants to perform in startBundle depend > on side inputs (and, implicitly, the window). For example, one might want to > initialize a (non-serializable) stateful object. In particular, this leads to > users incorrectly (in the case of triggered or non-globally-windowed side > inputs) memoizing this computation in the first processElement call. > One option would be to fold this into a customizable ViewFn. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-260) WindowMappingFn: Know the getSideInputWindow upper bound to release side input resources
[ https://issues.apache.org/jira/browse/BEAM-260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-260: - Summary: WindowMappingFn: Know the getSideInputWindow upper bound to release side input resources (was: Know the getSideInputWindow upper bound so can gc side input state) > WindowMappingFn: Know the getSideInputWindow upper bound to release side > input resources > > > Key: BEAM-260 > URL: https://issues.apache.org/jira/browse/BEAM-260 > Project: Beam > Issue Type: Bug > Components: beam-model, beam-model-runner-api >Reporter: Mark Shields >Assignee: Kenneth Knowles > > We currently have no static knowledge about the getSideInputWindow function, > and runners are thus forced to hold on to all side input state / elements in > case a future element reaches back into an earlier side input element. > Maybe we need an upper bound on lag from current to result of > getSideInputWindow so we can have a progressing gc horizon as we do for GKB > window state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1196) Serialize/deserialize Pipeline/TransformHierarchy to JSON
Kenneth Knowles created BEAM-1196: - Summary: Serialize/deserialize Pipeline/TransformHierarchy to JSON Key: BEAM-1196 URL: https://issues.apache.org/jira/browse/BEAM-1196 Project: Beam Issue Type: New Feature Components: beam-model-runner-api, sdk-java-core Reporter: Kenneth Knowles There are two sketches of a concrete format for a Pipeline: 1. The Avro schema in the design doc at https://s.apache.org/beam-runner-api 2. The JSON schema at https://github.com/apache/incubator-beam/pull/662 These should be pushed all the way through and added to the Java SDK. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1195) Give triggers URNs / JSON formats
Kenneth Knowles created BEAM-1195:
-
Summary: Give triggers URNs / JSON formats
Key: BEAM-1195
URL: https://issues.apache.org/jira/browse/BEAM-1195
Project: Beam
Issue Type: New Feature
Components: beam-model-runner-api
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
We have recently gotten to the point where triggers are just syntax, but it is
still shipped via Java serialization. To make it language-independent, we need
a concrete syntax.
Something like the following is fairly concise, tag adjacent to payload. I
haven't bothered making up fully verbose/namespaced URNs here.
{code}
{
"$urn": "OrFinally",
"main": {
"$urn": "EndOfWindow",
"early":
},
"finally": {
"$urn": "AfterCount",
"count": 45
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-1192) Give transforms URNs, use them instead of instanceof checks
[ https://issues.apache.org/jira/browse/BEAM-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-1192: -- Summary: Give transforms URNs, use them instead of instanceof checks (was: Add URNs to known transforms instead of using instanceof checks) > Give transforms URNs, use them instead of instanceof checks > --- > > Key: BEAM-1192 > URL: https://issues.apache.org/jira/browse/BEAM-1192 > Project: Beam > Issue Type: Improvement > Components: beam-model-runner-api > Reporter: Kenneth Knowles > > In the [Beam Runner AP|https://s.apache.org/beam-runner-api], transforms of > interest to runners are to be identified by URN. > Currently, Java-based runners use `instanceof` checks to both override > transforms and to implement primitive transforms. This language and > SDK-specific behavior should be replaced by adding these URNs, and checking > them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1192) Give transforms URNs, use them instead of instanceof checks
[ https://issues.apache.org/jira/browse/BEAM-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-1192: -- Issue Type: New Feature (was: Improvement) > Give transforms URNs, use them instead of instanceof checks > --- > > Key: BEAM-1192 > URL: https://issues.apache.org/jira/browse/BEAM-1192 > Project: Beam > Issue Type: New Feature > Components: beam-model-runner-api > Reporter: Kenneth Knowles > > In the [Beam Runner AP|https://s.apache.org/beam-runner-api], transforms of > interest to runners are to be identified by URN. > Currently, Java-based runners use `instanceof` checks to both override > transforms and to implement primitive transforms. This language and > SDK-specific behavior should be replaced by adding these URNs, and checking > them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1193) Give Coders URNs and document their binary formats outside the Java code base
Kenneth Knowles created BEAM-1193: - Summary: Give Coders URNs and document their binary formats outside the Java code base Key: BEAM-1193 URL: https://issues.apache.org/jira/browse/BEAM-1193 Project: Beam Issue Type: New Feature Components: beam-model, beam-model-runner-api Reporter: Kenneth Knowles Assignee: Frances Perry -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1192) Add URNs to known transforms instead of using instanceof checks
Kenneth Knowles created BEAM-1192: - Summary: Add URNs to known transforms instead of using instanceof checks Key: BEAM-1192 URL: https://issues.apache.org/jira/browse/BEAM-1192 Project: Beam Issue Type: Improvement Components: beam-model-runner-api Reporter: Kenneth Knowles In the [Beam Runner AP|https://s.apache.org/beam-runner-api], transforms of interest to runners are to be identified by URN. Currently, Java-based runners use `instanceof` checks to both override transforms and to implement primitive transforms. This language and SDK-specific behavior should be replaced by adding these URNs, and checking them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1191) Eliminate OldDoFn from the SDK
Kenneth Knowles created BEAM-1191:
-
Summary: Eliminate OldDoFn from the SDK
Key: BEAM-1191
URL: https://issues.apache.org/jira/browse/BEAM-1191
Project: Beam
Issue Type: Improvement
Components: sdk-java-core
Reporter: Kenneth Knowles
Priority: Minor
We are far enough along that now {{OldDoFn}} is not usable by users and
BEAM-498 is closed out. The remaining occurrences are limited to runners and
things that should end up in runners-core.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (BEAM-498) Make DoFnWithContext the new DoFn
[ https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-498. -- Resolution: Fixed Fix Version/s: 0.3.0-incubating > Make DoFnWithContext the new DoFn > - > > Key: BEAM-498 > URL: https://issues.apache.org/jira/browse/BEAM-498 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core > Reporter: Kenneth Knowles > Assignee: Kenneth Knowles > Labels: backward-incompatible > Fix For: 0.3.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1067) apex.examples.WordCountTest.testWordCountExample is flaky
[
https://issues.apache.org/jira/browse/BEAM-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-1067:
--
Summary: apex.examples.WordCountTest.testWordCountExample is flaky (was:
apex.examples.WordCountTest.testWordCountExample is be flaky)
> apex.examples.WordCountTest.testWordCountExample is flaky
> -
>
> Key: BEAM-1067
> URL: https://issues.apache.org/jira/browse/BEAM-1067
> Project: Beam
> Issue Type: Bug
> Components: runner-apex
>Reporter: Stas Levin
>Assignee: Thomas Weise
>
> Seems that
> {{org.apache.beam.runners.apex.examples.WordCountTest.testWordCountExample}}
> is flaky.
> For example,
> [this|https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/5408/org.apache.beam$beam-runners-apex/testReport/org.apache.beam.runners.apex.examples/WordCountTest/testWordCountExample/
> ] run failed although no changes were made in {{runner-apex}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (BEAM-1189) Add guide for release verifiers in the release guide
Kenneth Knowles created BEAM-1189: - Summary: Add guide for release verifiers in the release guide Key: BEAM-1189 URL: https://issues.apache.org/jira/browse/BEAM-1189 Project: Beam Issue Type: Improvement Components: website Reporter: Kenneth Knowles Assignee: James Malone This came up during the 0.4.0-incubating release discussion. There is this checklist: http://incubator.apache.org/guides/releasemanagement.html#check-list And we could point to that but make more detailed Beam-specific instructions on http://beam.apache.org/contribute/release-guide/#vote-on-the-release-candidate And the template for the vote email should include a link to suggested verification steps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (BEAM-1186) Migrate the remaining tests to use TestPipeline as a JUnit rule.
[
https://issues.apache.org/jira/browse/BEAM-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reopened BEAM-1186:
---
> Migrate the remaining tests to use TestPipeline as a JUnit rule.
>
>
> Key: BEAM-1186
> URL: https://issues.apache.org/jira/browse/BEAM-1186
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Stas Levin
>Assignee: Stas Levin
>Priority: Minor
> Fix For: 0.5.0-incubating
>
>
> Following up on [BEAM-1176|https://issues.apache.org/jira/browse/BEAM-1176],
> the following tests still have direct calls to {{TestPipeline.create()}}:
> * {{AvroIOGeneratedClassTest#runTestRead}}
> * {{ApproximateUniqueTest#runApproximateUniqueWithDuplicates}}
> * {{ApproximateUniqueTest#runApproximateUniqueWithSkewedDistributions}}
> * {{SampleTest#runPickAnyTest}}
> * {{BigtableIOTest#runReadTest}}
> Consider using [parametrised
> tests|https://github.com/Pragmatists/junitparams] as suggested by [~lcwik].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (BEAM-1176) Make our test suites use @Rule TestPipeline
[
https://issues.apache.org/jira/browse/BEAM-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-1176.
---
Resolution: Fixed
Fix Version/s: 0.5.0-incubating
> Make our test suites use @Rule TestPipeline
> ---
>
> Key: BEAM-1176
> URL: https://issues.apache.org/jira/browse/BEAM-1176
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
> Reporter: Kenneth Knowles
>Assignee: Stas Levin
>Priority: Minor
> Fix For: 0.5.0-incubating
>
>
> Now that [~staslev] has made {{TestPipeline}} a JUnit rule that performs
> useful sanity checks, we should port all of our tests to it so that they set
> a good example for users. Maybe we'll even catch some straggling tests with
> errors :-)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (BEAM-1186) Migrate the remaining tests to use TestPipeline as a JUnit rule.
[
https://issues.apache.org/jira/browse/BEAM-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-1186.
---
Resolution: Fixed
Fix Version/s: 0.5.0-incubating
> Migrate the remaining tests to use TestPipeline as a JUnit rule.
>
>
> Key: BEAM-1186
> URL: https://issues.apache.org/jira/browse/BEAM-1186
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Stas Levin
>Assignee: Stas Levin
>Priority: Minor
> Fix For: 0.5.0-incubating
>
>
> Following up on [BEAM-1176|https://issues.apache.org/jira/browse/BEAM-1176],
> the following tests still have direct calls to {{TestPipeline.create()}}:
> * {{AvroIOGeneratedClassTest#runTestRead}}
> * {{ApproximateUniqueTest#runApproximateUniqueWithDuplicates}}
> * {{ApproximateUniqueTest#runApproximateUniqueWithSkewedDistributions}}
> * {{SampleTest#runPickAnyTest}}
> * {{BigtableIOTest#runReadTest}}
> Consider using [parametrised
> tests|https://github.com/Pragmatists/junitparams] as suggested by [~lcwik].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-1067) apex.examples.WordCountTest.testWordCountExample is be flaky
[
https://issues.apache.org/jira/browse/BEAM-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-1067:
--
Summary: apex.examples.WordCountTest.testWordCountExample is be flaky
(was: apex.examples.WordCountTest.testWordCountExample may be flaky)
> apex.examples.WordCountTest.testWordCountExample is be flaky
>
>
> Key: BEAM-1067
> URL: https://issues.apache.org/jira/browse/BEAM-1067
> Project: Beam
> Issue Type: Bug
> Components: runner-apex
>Reporter: Stas Levin
>Assignee: Thomas Weise
>
> Seems that
> {{org.apache.beam.runners.apex.examples.WordCountTest.testWordCountExample}}
> is flaky.
> For example,
> [this|https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/5408/org.apache.beam$beam-runners-apex/testReport/org.apache.beam.runners.apex.examples/WordCountTest/testWordCountExample/
> ] run failed although no changes were made in {{runner-apex}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-440) Create.values() returns a type-unsafe Coder
[
https://issues.apache.org/jira/browse/BEAM-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-440:
-
Assignee: Jason White
> Create.values() returns a type-unsafe Coder
> ---
>
> Key: BEAM-440
> URL: https://issues.apache.org/jira/browse/BEAM-440
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Jason White
> Labels: newbie, starter
>
> {{Create.values()}} with no arguments will default to a {{VoidCoder}}, unless
> one is set later with {{setCoder(Coder)}}.
> Although it will encode its input correctly, this seems like a bad choice in
> many cases. E.g., with {{Flatten}}:
> {code}
> PCollection<KV<SomeClass, Integer>> initial = p.apply("First",
> Create.<KV<SomeClass, Integer>>of());
> PCollection<KV<SomeClass, Integer>> second =
> p.apply("Second", Create.of("a", "b")).apply(ParDo.of(new
> MyAvroDoFn()));
> PCollectionList
> .of(initial).and(second)
> .apply(Flatten.<KV<SomeClass, Integer>>pCollections());
> {code}
> This crashes trying to cast a KV from "Second" to a Void.class.
> 1. Suggest throwing a warning in {{getDefaultOutputCoder}} when defaulting to
> {{VoidCoder}} for an empty elements list. Should this be an error?
> 2. Suggest adding something like {{Create.empty(TypeDescriptor)}} to handle
> this case properly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-440) Create.values() returns a type-unsafe Coder
[
https://issues.apache.org/jira/browse/BEAM-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760233#comment-15760233
]
Kenneth Knowles commented on BEAM-440:
--
The root issue here is actually worse. There are two facets.
1. {{Create}} based on the value is unsafe in general. (full disclosure: I
implemented this unsafe "feature"). It adds a coder that is safe for _those
values only_ but not for their static type.
{code}
PCollection allStrings = pipeline.apply("Create strings",
Create.of("hello"));
{code}
The coder inference infrastructure reasonably assumes that
{{allStrings.getCoder()}} returns a coder that can handle all values of type
{{Object}}, which is as false in this example as it is for an empty {{Create}}.
Transforms like {{MapElements<Object, Object>}} will propagate the coder for
{{Object}} which will break if any non-string is in the output (which obviously
it could be).
2. {{Flatten}} quite explicitly assumes the same thing and grabs the first
coder it sees, so when you flatten a bunch of {{PCollection}} together
that have been made with {{Create}} you are in trouble.
So for this issue, we should probably just remove
https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Create.java#L496.
At the time we added it, it seemed handy to infer a coder based on the values,
since we didn't have any other information to go on due to the limits of Java
reflection, but the idea doesn't work without further validation which requires
as much user input as just setting the coder or type descriptor anyhow.
> Create.values() returns a type-unsafe Coder
> ---
>
> Key: BEAM-440
> URL: https://issues.apache.org/jira/browse/BEAM-440
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Daniel Halperin
> Labels: newbie, starter
>
> {{Create.values()}} with no arguments will default to a {{VoidCoder}}, unless
> one is set later with {{setCoder(Coder)}}.
> Although it will encode its input correctly, this seems like a bad choice in
> many cases. E.g., with {{Flatten}}:
> {code}
> PCollection<KV<SomeClass, Integer>> initial = p.apply("First",
> Create.<KV<SomeClass, Integer>>of());
> PCollection<KV<SomeClass, Integer>> second =
> p.apply("Second", Create.of("a", "b")).apply(ParDo.of(new
> MyAvroDoFn()));
> PCollectionList
> .of(initial).and(second)
> .apply(Flatten.<KV<SomeClass, Integer>>pCollections());
> {code}
> This crashes trying to cast a KV from "Second" to a Void.class.
> 1. Suggest throwing a warning in {{getDefaultOutputCoder}} when defaulting to
> {{VoidCoder}} for an empty elements list. Should this be an error?
> 2. Suggest adding something like {{Create.empty(TypeDescriptor)}} to handle
> this case properly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-440) Create.values() returns a type-unsafe Coder
[
https://issues.apache.org/jira/browse/BEAM-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-440:
-
Description:
{{Create.values()}} with no arguments will default to a {{VoidCoder}}, unless
one is set later with {{setCoder(Coder)}}.
Although it will encode its input correctly, this seems like a bad choice in
many cases. E.g., with {{Flatten}}:
{code}
PCollection<KV<SomeClass, Integer>> initial = p.apply("First",
Create.<KV<SomeClass, Integer>>of());
PCollection<KV<SomeClass, Integer>> second =
p.apply("Second", Create.of("a", "b")).apply(ParDo.of(new
MyAvroDoFn()));
PCollectionList
.of(initial).and(second)
.apply(Flatten.<KV<SomeClass, Integer>>pCollections());
{code}
This crashes trying to cast a KV from "Second" to a Void.class.
1. Suggest throwing a warning in {{getDefaultOutputCoder}} when defaulting to
{{VoidCoder}} for an empty elements list. Should this be an error?
2. Suggest adding something like {{Create.empty(TypeDescriptor)}} to handle
this case properly.
was:
Create.values() with no arguments will default to a VoidCoder, unless one is
set later with #setCoder(Coder).
Although it will encode its input correctly, this seems like a bad choice in
many cases. E.g., with Flatten:
PCollection<KV<SomeClass, Integer>> initial = p.apply("First",
Create.<KV<SomeClass, Integer>>of());
PCollection<KV<SomeClass, Integer>> second =
p.apply("Second", Create.of("a", "b")).apply(ParDo.of(new
MyAvroDoFn()));
PCollectionList
.of(initial).and(second)
.apply(Flatten.<KV<SomeClass, Integer>>pCollections());
This crashes trying to cast a KV from "Second" to a Void.class.
1. Suggest throwing a warning in #getDefaultOutputCoder when defaulting to
VoidCoder for an empty elements list. Should this be an error?
2. Suggest adding something like Create.empty(TypeDescriptor) to handle this
case properly.
> Create.values() returns a type-unsafe Coder
> ---
>
> Key: BEAM-440
> URL: https://issues.apache.org/jira/browse/BEAM-440
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Daniel Halperin
> Labels: newbie, starter
>
> {{Create.values()}} with no arguments will default to a {{VoidCoder}}, unless
> one is set later with {{setCoder(Coder)}}.
> Although it will encode its input correctly, this seems like a bad choice in
> many cases. E.g., with {{Flatten}}:
> {code}
> PCollection<KV<SomeClass, Integer>> initial = p.apply("First",
> Create.<KV<SomeClass, Integer>>of());
> PCollection<KV<SomeClass, Integer>> second =
> p.apply("Second", Create.of("a", "b")).apply(ParDo.of(new
> MyAvroDoFn()));
> PCollectionList
> .of(initial).and(second)
> .apply(Flatten.<KV<SomeClass, Integer>>pCollections());
> {code}
> This crashes trying to cast a KV from "Second" to a Void.class.
> 1. Suggest throwing a warning in {{getDefaultOutputCoder}} when defaulting to
> {{VoidCoder}} for an empty elements list. Should this be an error?
> 2. Suggest adding something like {{Create.empty(TypeDescriptor)}} to handle
> this case properly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-210) Allow control of empty ON_TIME panes analogous to final panes
[ https://issues.apache.org/jira/browse/BEAM-210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-210: - Component/s: (was: runner-core) sdk-java-core beam-model-runner-api beam-model > Allow control of empty ON_TIME panes analogous to final panes > - > > Key: BEAM-210 > URL: https://issues.apache.org/jira/browse/BEAM-210 > Project: Beam > Issue Type: Bug > Components: beam-model, beam-model-runner-api, sdk-java-core >Reporter: Mark Shields >Assignee: Thomas Groh > > Today, ON_TIME panes are emitted whether or not they are empty. We had > decided that for final panes the user would want to control this behavior, to > control data volume. But for ON_TIME panes no such control exists. The > rationale is perhaps that the ON_TIME pane is a fundamental result that > should not be elided. To be considered: whether this is what we want. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-885) Move PipelineOptions from Pipeline.create() to Pipeline.run()
[ https://issues.apache.org/jira/browse/BEAM-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-885: - Component/s: beam-model-runner-api > Move PipelineOptions from Pipeline.create() to Pipeline.run() > - > > Key: BEAM-885 > URL: https://issues.apache.org/jira/browse/BEAM-885 > Project: Beam > Issue Type: New Feature > Components: beam-model-runner-api, sdk-java-core >Reporter: Thomas Groh >Assignee: Thomas Groh > Labels: backward-incompatible > > The specification of a Pipeline should be independent of its PipelineOptions. > This delays specification of the options, including choices like Pipeline > Runner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-646) Get runners out of the apply()
[ https://issues.apache.org/jira/browse/BEAM-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-646: - Component/s: beam-model-runner-api > Get runners out of the apply() > -- > > Key: BEAM-646 > URL: https://issues.apache.org/jira/browse/BEAM-646 > Project: Beam > Issue Type: Improvement > Components: beam-model-runner-api, sdk-java-core > Reporter: Kenneth Knowles >Assignee: Thomas Groh > > Right now, the runner intercepts calls to apply() and replaces transforms as > we go. This means that there is no "original" user graph. For portability and > misc architectural benefits, we would like to build the original graph first, > and have the runner override later. > Some runners already work in this manner, but we could integrate it more > smoothly, with more validation, via some handy APIs on e.g. the Pipeline > object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-260) Know the getSideInputWindow upper bound so can gc side input state
[ https://issues.apache.org/jira/browse/BEAM-260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-260: - Component/s: beam-model-runner-api > Know the getSideInputWindow upper bound so can gc side input state > -- > > Key: BEAM-260 > URL: https://issues.apache.org/jira/browse/BEAM-260 > Project: Beam > Issue Type: Bug > Components: beam-model, beam-model-runner-api >Reporter: Mark Shields > Assignee: Kenneth Knowles > > We currently have no static knowledge about the getSideInputWindow function, > and runners are thus forced to hold on to all side input state / elements in > case a future element reaches back into an earlier side input element. > Maybe we need an upper bound on lag from current to result of > getSideInputWindow so we can have a progressing gc horizon as we do for GKB > window state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-742) Move trigger state machines to runners-core, convert triggers to AST
[ https://issues.apache.org/jira/browse/BEAM-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-742: - Component/s: beam-model-runner-api > Move trigger state machines to runners-core, convert triggers to AST > > > Key: BEAM-742 > URL: https://issues.apache.org/jira/browse/BEAM-742 > Project: Beam > Issue Type: Sub-task > Components: beam-model-runner-api, runner-core, sdk-java-core > Reporter: Kenneth Knowles > Assignee: Kenneth Knowles > Fix For: 0.4.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-846) Decouple side input window mapping from WindowFn
[ https://issues.apache.org/jira/browse/BEAM-846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-846: - Component/s: beam-model-runner-api > Decouple side input window mapping from WindowFn > > > Key: BEAM-846 > URL: https://issues.apache.org/jira/browse/BEAM-846 > Project: Beam > Issue Type: New Feature > Components: beam-model, beam-model-runner-api, sdk-java-core >Reporter: Robert Bradshaw > Assignee: Kenneth Knowles > Labels: backward-incompatible > > Currently the main WindowFn provides as getSideInputWindow method. Instead, > this mapping should be specified per-side-input (thought the default mapping > would remain the same). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1159) DoFnSignature should have info on the fn's side inputs and outputs
[ https://issues.apache.org/jira/browse/BEAM-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760080#comment-15760080 ] Kenneth Knowles commented on BEAM-1159: --- The expected resolution of this ticket was actually designed long ago, and is included as an appendix to the new DoFn design, here: https://s.apache.org/a-new-dofn#heading=h.1budnm7l01ko It hasn't been a priority yet, but it isn't difficult. It can probably be done in a backwards-compatible manner, though it would be cleaner if we have time to add the new form of support and remove the old way. > DoFnSignature should have info on the fn's side inputs and outputs > -- > > Key: BEAM-1159 > URL: https://issues.apache.org/jira/browse/BEAM-1159 > Project: Beam > Issue Type: Improvement >Reporter: Eugene Kirpichov > > This is logically part of the fn itself, rather than its enclosing transform. > Example where this would have been important: > signature.processElement().observesWindow() should return true for a DoFn > that has any side inputs, since side inputs are windowed. See BEAM-1149. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-1149) Side input access fails in direct runner (possibly others too) when input element in multiple windows
[
https://issues.apache.org/jira/browse/BEAM-1149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-1149.
---
Resolution: Fixed
> Side input access fails in direct runner (possibly others too) when input
> element in multiple windows
> -
>
> Key: BEAM-1149
> URL: https://issues.apache.org/jira/browse/BEAM-1149
> Project: Beam
> Issue Type: Bug
>Reporter: Eugene Kirpichov
> Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 0.4.0-incubating
>
>
> {code:java}
> private static class FnWithSideInputs extends DoFn<String, String> {
> private final PCollectionView view;
> private FnWithSideInputs(PCollectionView view) {
> this.view = view;
> }
> @ProcessElement
> public void processElement(ProcessContext c) {
> c.output(c.element() + ":" + c.sideInput(view));
> }
> }
> @Test
> public void testSideInputsWithMultipleWindows() {
> Pipeline p = TestPipeline.create();
> MutableDateTime mutableNow = Instant.now().toMutableDateTime();
> mutableNow.setMillisOfSecond(0);
> Instant now = mutableNow.toInstant();
> SlidingWindows windowFn =
>
> SlidingWindows.of(Duration.standardSeconds(5)).every(Duration.standardSeconds(1));
> PCollectionView view =
> p.apply(Create.of(1)).apply(View.asSingleton());
> PCollection res =
> p.apply(Create.timestamped(TimestampedValue.of("a", now)))
> .apply(Window.into(windowFn))
> .apply(ParDo.of(new FnWithSideInputs(view)).withSideInputs(view));
> PAssert.that(res).containsInAnyOrder("a:1");
> p.run();
> }
> {code}
> This fails with the following exception:
> {code}
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.IllegalStateException: sideInput called when main input element is
> in multiple windows
> at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:343)
> at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:1)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:176)
> at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:112)
> at
> Caused by: java.lang.IllegalStateException: sideInput called when main input
> element is in multiple windows
> at
> org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.sideInput(SimpleDoFnRunner.java:514)
> at
> org.apache.beam.sdk.transforms.ParDoTest$FnWithSideInputs.processElement(ParDoTest.java:738)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-1164) Allow a DoFn to opt in to mutating it's input
[
https://issues.apache.org/jira/browse/BEAM-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760074#comment-15760074
]
Kenneth Knowles commented on BEAM-1164:
---
I think with the new {{DoFn}} there is a fairly elegant solution here.
Today we write:
{code}
new DoFn<Foo, Baz>() {
@ProcessElement
public void processElem(ProcessContext ctx) {
... ctx.element() ...
}
}
{code}
We'd like to allow the user to request only the element, both for clarity and
for potential optimization, as in
{code}
new DoFn<Foo, Baz>() {
@ProcessElement
public void processElem(Element elem) {
... elem.get() ...
}
}
{code}
where {{Element}} is a distinguished inner class, to avoid repeating verbose
input types.
>From here, it is a short step to saying that you want a mutable element:
{code}
new DoFn<Foo, Baz>() {
@ProcessElement
public void processElem(MutableElement elem) {
... elem.get().setBizzle(...) ...
}
}
{code}
At the level of the "JSON" runner API, we will need to tag the user-defined
function with the fact that it intends to mutate its input. The Java SDK will
analyze the method signature, as usual, to discern this automatically.
A runner will then be free to decide between disabling optimizations or cloning
elements when necessary.
> Allow a DoFn to opt in to mutating it's input
> -
>
> Key: BEAM-1164
> URL: https://issues.apache.org/jira/browse/BEAM-1164
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Reporter: Frances Perry
>Priority: Minor
>
> Runners generally can't tell if a DoFn is mutating inputs, but assuming so by
> default leads to significant performance implications from unnecessary
> copying (around sibling fusion, etc). So instead the model prevents mutating
> inputs, and the Direct Runner validates this behavior. (See:
> http://beam.incubator.apache.org/contribute/design-principles/#make-efficient-things-easy-rather-than-make-easy-things-efficient)
>
> However, if users are processing a small number of large records by making
> incremental changes (for example, genomics use cases), the cost of
> immutability requirement can be very large. As a workaround, users sometimes
> do suboptimal things (fusing ParDos by hand) or undefined things when they
> expect the immutability requirement is unnecessarily strict (adding no-op
> coders in places they hope the runner won't be materializing things, mutating
> things anyway when they don't expect sibling fusion to happen, etc).
> We should consider adding a signal (MutatingDoFn?) that users explicitly opt
> in to to say their code may mutate inputs. The runner can then use this
> assumption to either prevent optimizations that would break in the face of
> this or insert additional copies as needed to allow optimizations to preserve
> semantics.
> See this related user@ discussion:
> https://lists.apache.org/thread.html/f39689f54147117f3fc54c498eff1a20fa73f1be5b5cad5b6f816fd3@%3Cuser.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-651) Making TypedPValue.setTypeDescriptorInternal no longer Internal
[ https://issues.apache.org/jira/browse/BEAM-651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-651: - Summary: Making TypedPValue.setTypeDescriptorInternal no longer Internal (was: Consider making TypedPValue.setTypeDescriptorInternal no longer Internal) > Making TypedPValue.setTypeDescriptorInternal no longer Internal > --- > > Key: BEAM-651 > URL: https://issues.apache.org/jira/browse/BEAM-651 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core > Reporter: Kenneth Knowles >Assignee: Neelesh Srinivas Salian >Priority: Minor > Labels: easy, easyfix, starter > Fix For: 0.4.0-incubating > > > This would give fairly pithy answers to StackOverflow questions sometimes. > When choosing between .getOutputCoder() and .getOutputTypeDescriptor() for a > transform/DoFn we often choose the type, so the coder registry can do its > thing. > This would also give a similar choice between .setCoder(...) and > .setTypeDescriptor(...). > And anyhow we have the intention of removing our practice of the "*Internal" > suffix, so this one might be most easily solved by making it public. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-651) Add public TypedPValue.setTypeDescriptor
[ https://issues.apache.org/jira/browse/BEAM-651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-651: - Summary: Add public TypedPValue.setTypeDescriptor (was: Making TypedPValue.setTypeDescriptorInternal no longer Internal) > Add public TypedPValue.setTypeDescriptor > > > Key: BEAM-651 > URL: https://issues.apache.org/jira/browse/BEAM-651 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core > Reporter: Kenneth Knowles >Assignee: Neelesh Srinivas Salian >Priority: Minor > Labels: easy, easyfix, starter > Fix For: 0.4.0-incubating > > > This would give fairly pithy answers to StackOverflow questions sometimes. > When choosing between .getOutputCoder() and .getOutputTypeDescriptor() for a > transform/DoFn we often choose the type, so the coder registry can do its > thing. > This would also give a similar choice between .setCoder(...) and > .setTypeDescriptor(...). > And anyhow we have the intention of removing our practice of the "*Internal" > suffix, so this one might be most easily solved by making it public. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-35) Remove timer multiplexing and give timers identifiers
[ https://issues.apache.org/jira/browse/BEAM-35?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-35: Issue Type: Improvement (was: New Feature) > Remove timer multiplexing and give timers identifiers > - > > Key: BEAM-35 > URL: https://issues.apache.org/jira/browse/BEAM-35 > Project: Beam > Issue Type: Improvement > Components: runner-core > Reporter: Kenneth Knowles > Assignee: Kenneth Knowles > Fix For: 0.4.0-incubating > > > Today, timers set by the runner core of the SDK are identified by timestamp, > so multiple timers for the same time result in only one backend timer. This > is runner-specific and obsolete, and makes it unsafe to support timer > deletion because backend timers have no real owner. User-facing timers will > require explicit identifiers anyhow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-864) RAT check fails on DEPENDENCIES
[ https://issues.apache.org/jira/browse/BEAM-864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-864: - Summary: RAT check fails on DEPENDENCIES (was: Update to latest Apache Maven-Parent) > RAT check fails on DEPENDENCIES > --- > > Key: BEAM-864 > URL: https://issues.apache.org/jira/browse/BEAM-864 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Aljoscha Krettek >Assignee: Jean-Baptiste Onofré > Fix For: 0.4.0-incubating > > > The release plugin creates a DEPENDENCIES file that is not properly excluded > from the rat check with the current version of the Apache Maven-Parent that > we are using. > This is the relevant RAT Jira issue: > https://issues.apache.org/jira/browse/RAT-184 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-808) Increase "spark.port.maxRetries" to avoid BindException in ROS.
[ https://issues.apache.org/jira/browse/BEAM-808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-808: - Issue Type: Bug (was: Improvement) > Increase "spark.port.maxRetries" to avoid BindException in ROS. > --- > > Key: BEAM-808 > URL: https://issues.apache.org/jira/browse/BEAM-808 > Project: Beam > Issue Type: Bug > Components: runner-spark >Reporter: Amit Sela >Assignee: Amit Sela > Fix For: 0.4.0-incubating > > > The default is 16, and it's hard to know which port failed to bind since > there's no logging. > Let's start with 64. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-830) Launcher for ApexRunner execution on YARN cluster
[ https://issues.apache.org/jira/browse/BEAM-830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-830: - Issue Type: New Feature (was: Improvement) > Launcher for ApexRunner execution on YARN cluster > -- > > Key: BEAM-830 > URL: https://issues.apache.org/jira/browse/BEAM-830 > Project: Beam > Issue Type: New Feature > Components: runner-apex >Reporter: Thomas Weise >Assignee: Thomas Weise > Fix For: 0.4.0-incubating > > > Currently the ApexRunner only support execution in embedded mode. Add the > support to package the dependencies and run the Apex app on a YARN cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1060) Make DoFnTester use new DoFn
[ https://issues.apache.org/jira/browse/BEAM-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-1060: -- Issue Type: New Feature (was: Improvement) > Make DoFnTester use new DoFn > > > Key: BEAM-1060 > URL: https://issues.apache.org/jira/browse/BEAM-1060 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov > Fix For: 0.4.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-252) Make Regex Transform
[ https://issues.apache.org/jira/browse/BEAM-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-252: - Issue Type: New Feature (was: Improvement) > Make Regex Transform > > > Key: BEAM-252 > URL: https://issues.apache.org/jira/browse/BEAM-252 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Jesse Anderson >Assignee: Jesse Anderson > Fix For: 0.4.0-incubating > > > There needs to be an easier way to run Regular Expressions as part of a > transform. This will make string-based ETL much easier. > The transform should support using the matches and find methods. The > transform should allow you to choose a group in the regex to output. The > transform should allow single strings to be output or KV's of strings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1060) Make DoFnTester use new DoFn
[ https://issues.apache.org/jira/browse/BEAM-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-1060: -- Issue Type: Improvement (was: New Feature) > Make DoFnTester use new DoFn > > > Key: BEAM-1060 > URL: https://issues.apache.org/jira/browse/BEAM-1060 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov > Fix For: 0.4.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-939) New credentials code broke Dataflow runner
[
https://issues.apache.org/jira/browse/BEAM-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-939:
-
Issue Type: Bug (was: New Feature)
> New credentials code broke Dataflow runner
> --
>
> Key: BEAM-939
> URL: https://issues.apache.org/jira/browse/BEAM-939
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-gcp
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_MavenVerify/1753/
> {code}
> java.lang.NoSuchMethodError:
> com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(Lcom/google/api/client/http/HttpTransport;)Lcom/google/auth/oauth2/GoogleCredentials;
> at
> com.google.cloud.bigtable.config.CredentialFactory.getApplicationDefaultCredential(CredentialFactory.java:207)
> at
> com.google.cloud.bigtable.config.CredentialFactory.getCredentials(CredentialFactory.java:112)
> at
> com.google.cloud.bigtable.grpc.io.CredentialInterceptorCache.getCredentialsInterceptor(CredentialInterceptorCache.java:94)
> at
> com.google.cloud.bigtable.grpc.BigtableSession.(BigtableSession.java:272)
> at
> org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.tableExists(BigtableServiceImpl.java:81)
> at
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:296)
> at
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:185)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:399)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
> at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:47)
> at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:158)
> at
> org.apache.beam.sdk.io.gcp.bigtable.BigtableReadIT.testE2EBigtableRead(BigtableReadIT.java:53)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-651) Consider making TypedPValue.setTypeDescriptorInternal no longer Internal
[ https://issues.apache.org/jira/browse/BEAM-651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-651: - Issue Type: New Feature (was: Wish) > Consider making TypedPValue.setTypeDescriptorInternal no longer Internal > > > Key: BEAM-651 > URL: https://issues.apache.org/jira/browse/BEAM-651 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core > Reporter: Kenneth Knowles >Assignee: Neelesh Srinivas Salian >Priority: Minor > Labels: easy, easyfix, starter > Fix For: 0.4.0-incubating > > > This would give fairly pithy answers to StackOverflow questions sometimes. > When choosing between .getOutputCoder() and .getOutputTypeDescriptor() for a > transform/DoFn we often choose the type, so the coder registry can do its > thing. > This would also give a similar choice between .setCoder(...) and > .setTypeDescriptor(...). > And anyhow we have the intention of removing our practice of the "*Internal" > suffix, so this one might be most easily solved by making it public. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-947) WindowedWordCountIT fails due to not setting "--output" pipeline options
[ https://issues.apache.org/jira/browse/BEAM-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-947: - Issue Type: Bug (was: New Feature) > WindowedWordCountIT fails due to not setting "--output" pipeline options > > > Key: BEAM-947 > URL: https://issues.apache.org/jira/browse/BEAM-947 > Project: Beam > Issue Type: Bug > Components: examples-java >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Blocker > Fix For: 0.4.0-incubating > > > Example failure: > https://builds.apache.org/job/beam_PreCommit_MavenVerify/4793/org.apache.beam$beam-examples-java/testReport/junit/org.apache.beam.examples/WindowedWordCountIT/testWindowedWordCountInStreaming/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-261) Apache Apex Runner
[ https://issues.apache.org/jira/browse/BEAM-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-261: - Issue Type: New Feature (was: Wish) > Apache Apex Runner > -- > > Key: BEAM-261 > URL: https://issues.apache.org/jira/browse/BEAM-261 > Project: Beam > Issue Type: New Feature > Components: runner-apex >Reporter: Suminda Dharmasena >Assignee: Thomas Weise > Fix For: 0.4.0-incubating > > > Like Spark, Flink and GearPump, Apache Apex also does have advantages. Is it > possible to have a runner for Apache Apex? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1176) Make our test suites use @Rule TestPipeline
Kenneth Knowles created BEAM-1176:
-
Summary: Make our test suites use @Rule TestPipeline
Key: BEAM-1176
URL: https://issues.apache.org/jira/browse/BEAM-1176
Project: Beam
Issue Type: Improvement
Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Stas Levin
Priority: Minor
Now that [~staslev] has made {{TestPipeline}} a JUnit rule that performs useful
sanity checks, we should port all of our tests to it so that they set a good
example for users. Maybe we'll even catch some straggling tests with errors :-)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (BEAM-85) PAssert needs sanity check that it's used correctly
[
https://issues.apache.org/jira/browse/BEAM-85?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-85.
-
Resolution: Fixed
Fix Version/s: 0.5.0-incubating
When used as a `@Rule` this is now the case. Now we need to port the tests,
which I'd say fits more into BEAM0-298 or potentially a new ticket.
> PAssert needs sanity check that it's used correctly
> ---
>
> Key: BEAM-85
> URL: https://issues.apache.org/jira/browse/BEAM-85
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Stas Levin
> Fix For: 0.5.0-incubating
>
>
> We should validate two things:
> # DataflowAssert is not added to a pipeline that has already been run.
> # The pipeline is run after the DataflowAssert is added.
> If either of these are not validated, then it is possible that the test
> doesn't actually verify anything.
> This code should throw an assertion error or fail in some other way.
> {code}
> Pipeline p = TestPipeline.create();
> PCollection value = p.apply(Create.of(Boolean.FALSE));
> p.run();
> DataflowAssert.thatSingleton(value).isEqualTo(true);
> {code}
> but it would pass silently.
> similarly, this code wills pass silently:
> {code}
> Pipeline p = TestPipeline.create();
> PCollection value = p.apply(Create.of(Boolean.FALSE));
> DataflowAssert.thatSingleton(value).isEqualTo(true);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (BEAM-1175) Separate runners-core Javadoc from SDK Javadoc, and perhaps other artifacts
Kenneth Knowles created BEAM-1175: - Summary: Separate runners-core Javadoc from SDK Javadoc, and perhaps other artifacts Key: BEAM-1175 URL: https://issues.apache.org/jira/browse/BEAM-1175 Project: Beam Issue Type: Bug Components: website Reporter: Kenneth Knowles Assignee: James Malone Currently, the runners-core Javadoc is included with the SDK's Javadoc on the website. Pipeline authors should only be viewing the SDK's Javadoc, and likely the IOs. Generally, while I do think we can't really separate every IO Javadoc, it is extremely confusing to have Javadoc for separate POM dependencies included together, since the classes listed will not actually be available to someone who just depends on the SDK. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1172) Precommit integration tests should include some merging windows
Kenneth Knowles created BEAM-1172: - Summary: Precommit integration tests should include some merging windows Key: BEAM-1172 URL: https://issues.apache.org/jira/browse/BEAM-1172 Project: Beam Issue Type: Bug Components: testing Reporter: Kenneth Knowles Assignee: Kenneth Knowles -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1139) Investigate failures in precommit - Apex & Kryo
[ https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-1139: -- Assignee: (was: Kenneth Knowles) > Investigate failures in precommit - Apex & Kryo > --- > > Key: BEAM-1139 > URL: https://issues.apache.org/jira/browse/BEAM-1139 > Project: Beam > Issue Type: Improvement > Components: runner-apex > Reporter: Kenneth Knowles >Priority: Minor > > https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/ > This is not necessarily a bug in the Apex runner, but it looks like this > class cannot be serialized via Kryo while the Apex runner needs it to be. > Probably the fix is to roll-forwards a simple change to make it Kryo > serializable. > It is not clear to me the difference between this test run and others. > Clearly there is a coverage gap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (BEAM-1148) Port PAssert away from Aggregators
[ https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749192#comment-15749192 ] Kenneth Knowles edited comment on BEAM-1148 at 12/14/16 7:17 PM: - The trouble with side output + combine is that we don't have access to the result via PipelineResult (aka handle to the job). So I think there's actual design work here. was (Author: kenn): The trouble with side output + combine is that we don't have access to the result via PipelineResult. So I think there's actual design work here. > Port PAssert away from Aggregators > -- > > Key: BEAM-1148 > URL: https://issues.apache.org/jira/browse/BEAM-1148 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core > Reporter: Kenneth Knowles > > One step in the removal of Aggregators (in favor of Metrics) is to remove our > reliance on them for PAssert checking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1148) Port PAssert away from Aggregators
[ https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749192#comment-15749192 ] Kenneth Knowles commented on BEAM-1148: --- The trouble with side output + combine is that we don't have access to the result via PipelineResult. So I think there's actual design work here. > Port PAssert away from Aggregators > -- > > Key: BEAM-1148 > URL: https://issues.apache.org/jira/browse/BEAM-1148 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core > Reporter: Kenneth Knowles > > One step in the removal of Aggregators (in favor of Metrics) is to remove our > reliance on them for PAssert checking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (BEAM-1149) Side input access fails in direct runner (possibly others too) when input element in multiple windows
[
https://issues.apache.org/jira/browse/BEAM-1149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reopened BEAM-1149:
---
Assignee: Kenneth Knowles (was: Eugene Kirpichov)
Still failing Spark postcommit
https://builds.apache.org/job/beam_PostCommit_Java_RunnableOnService_Spark/409/org.apache.beam$beam-runners-spark/testReport/org.apache.beam.sdk.transforms/ParDoTest/testSideInputsWithMultipleWindows/
> Side input access fails in direct runner (possibly others too) when input
> element in multiple windows
> -
>
> Key: BEAM-1149
> URL: https://issues.apache.org/jira/browse/BEAM-1149
> Project: Beam
> Issue Type: Bug
>Reporter: Eugene Kirpichov
> Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 0.4.0-incubating
>
>
> {code:java}
> private static class FnWithSideInputs extends DoFn<String, String> {
> private final PCollectionView view;
> private FnWithSideInputs(PCollectionView view) {
> this.view = view;
> }
> @ProcessElement
> public void processElement(ProcessContext c) {
> c.output(c.element() + ":" + c.sideInput(view));
> }
> }
> @Test
> public void testSideInputsWithMultipleWindows() {
> Pipeline p = TestPipeline.create();
> MutableDateTime mutableNow = Instant.now().toMutableDateTime();
> mutableNow.setMillisOfSecond(0);
> Instant now = mutableNow.toInstant();
> SlidingWindows windowFn =
>
> SlidingWindows.of(Duration.standardSeconds(5)).every(Duration.standardSeconds(1));
> PCollectionView view =
> p.apply(Create.of(1)).apply(View.asSingleton());
> PCollection res =
> p.apply(Create.timestamped(TimestampedValue.of("a", now)))
> .apply(Window.into(windowFn))
> .apply(ParDo.of(new FnWithSideInputs(view)).withSideInputs(view));
> PAssert.that(res).containsInAnyOrder("a:1");
> p.run();
> }
> {code}
> This fails with the following exception:
> {code}
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.IllegalStateException: sideInput called when main input element is
> in multiple windows
> at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:343)
> at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:1)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:176)
> at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:112)
> at
> Caused by: java.lang.IllegalStateException: sideInput called when main input
> element is in multiple windows
> at
> org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.sideInput(SimpleDoFnRunner.java:514)
> at
> org.apache.beam.sdk.transforms.ParDoTest$FnWithSideInputs.processElement(ParDoTest.java:738)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-1149) Side input access fails in direct runner (possibly others too) when input element in multiple windows
[
https://issues.apache.org/jira/browse/BEAM-1149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-1149:
--
Assignee: Eugene Kirpichov
> Side input access fails in direct runner (possibly others too) when input
> element in multiple windows
> -
>
> Key: BEAM-1149
> URL: https://issues.apache.org/jira/browse/BEAM-1149
> Project: Beam
> Issue Type: Bug
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Blocker
>
> {code:java}
> private static class FnWithSideInputs extends DoFn<String, String> {
> private final PCollectionView view;
> private FnWithSideInputs(PCollectionView view) {
> this.view = view;
> }
> @ProcessElement
> public void processElement(ProcessContext c) {
> c.output(c.element() + ":" + c.sideInput(view));
> }
> }
> @Test
> public void testSideInputsWithMultipleWindows() {
> Pipeline p = TestPipeline.create();
> MutableDateTime mutableNow = Instant.now().toMutableDateTime();
> mutableNow.setMillisOfSecond(0);
> Instant now = mutableNow.toInstant();
> SlidingWindows windowFn =
>
> SlidingWindows.of(Duration.standardSeconds(5)).every(Duration.standardSeconds(1));
> PCollectionView view =
> p.apply(Create.of(1)).apply(View.asSingleton());
> PCollection res =
> p.apply(Create.timestamped(TimestampedValue.of("a", now)))
> .apply(Window.into(windowFn))
> .apply(ParDo.of(new FnWithSideInputs(view)).withSideInputs(view));
> PAssert.that(res).containsInAnyOrder("a:1");
> p.run();
> }
> {code}
> This fails with the following exception:
> {code}
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.IllegalStateException: sideInput called when main input element is
> in multiple windows
> at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:343)
> at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:1)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:176)
> at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:112)
> at
> Caused by: java.lang.IllegalStateException: sideInput called when main input
> element is in multiple windows
> at
> org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.sideInput(SimpleDoFnRunner.java:514)
> at
> org.apache.beam.sdk.transforms.ParDoTest$FnWithSideInputs.processElement(ParDoTest.java:738)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (BEAM-1150) Side inputs broken with sliding windows (or any multi-window WindowFn)
[ https://issues.apache.org/jira/browse/BEAM-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles closed BEAM-1150. - > Side inputs broken with sliding windows (or any multi-window WindowFn) > -- > > Key: BEAM-1150 > URL: https://issues.apache.org/jira/browse/BEAM-1150 > Project: Beam > Issue Type: Bug > Components: sdk-java-core > Reporter: Kenneth Knowles >Assignee: Eugene Kirpichov >Priority: Blocker > Fix For: Not applicable > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-1150) Side inputs broken with sliding windows (or any multi-window WindowFn)
[ https://issues.apache.org/jira/browse/BEAM-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-1150. --- Resolution: Duplicate Fix Version/s: Not applicable > Side inputs broken with sliding windows (or any multi-window WindowFn) > -- > > Key: BEAM-1150 > URL: https://issues.apache.org/jira/browse/BEAM-1150 > Project: Beam > Issue Type: Bug > Components: sdk-java-core > Reporter: Kenneth Knowles >Assignee: Eugene Kirpichov >Priority: Blocker > Fix For: Not applicable > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1150) Side inputs broken with sliding windows (or any multi-window WindowFn)
Kenneth Knowles created BEAM-1150: - Summary: Side inputs broken with sliding windows (or any multi-window WindowFn) Key: BEAM-1150 URL: https://issues.apache.org/jira/browse/BEAM-1150 Project: Beam Issue Type: Bug Components: sdk-java-core Reporter: Kenneth Knowles Assignee: Eugene Kirpichov Priority: Blocker -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1148) Port PAssert away from Aggregators
Kenneth Knowles created BEAM-1148: - Summary: Port PAssert away from Aggregators Key: BEAM-1148 URL: https://issues.apache.org/jira/browse/BEAM-1148 Project: Beam Issue Type: New Feature Components: sdk-java-core Reporter: Kenneth Knowles One step in the removal of Aggregators (in favor of Metrics) is to remove our reliance on them for PAssert checking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (BEAM-967) Add Jenkins postcommit for Apex runner RunnableOnService tests
[ https://issues.apache.org/jira/browse/BEAM-967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-967: Assignee: Kenneth Knowles (was: Jason Kuster) > Add Jenkins postcommit for Apex runner RunnableOnService tests > -- > > Key: BEAM-967 > URL: https://issues.apache.org/jira/browse/BEAM-967 > Project: Beam > Issue Type: Bug > Components: runner-apex, testing > Reporter: Kenneth Knowles > Assignee: Kenneth Knowles > > During development of the Apex runner, all RunnableOnService tests were run > in precommit. Now that it has been merged, they are not. It needs the usual > treatment that we give runners. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-967) Add Jenkins postcommit for Apex runner RunnableOnService tests
[ https://issues.apache.org/jira/browse/BEAM-967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746035#comment-15746035 ] Kenneth Knowles commented on BEAM-967: -- Since we now have access to Jenkins config via the DSL, I'll give it a try. > Add Jenkins postcommit for Apex runner RunnableOnService tests > -- > > Key: BEAM-967 > URL: https://issues.apache.org/jira/browse/BEAM-967 > Project: Beam > Issue Type: Bug > Components: runner-apex, testing > Reporter: Kenneth Knowles >Assignee: Jason Kuster > > During development of the Apex runner, all RunnableOnService tests were run > in precommit. Now that it has been merged, they are not. It needs the usual > treatment that we give runners. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1139) Investigate failures in precommit - Apex & Kryo
[ https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15745780#comment-15745780 ] Kenneth Knowles commented on BEAM-1139: --- This looks like it is actually a problem with our example pom and/or Jenkins invocation, where the Spark runner's deps override the Apex runner's deps. I'll take this and mess with the pom. > Investigate failures in precommit - Apex & Kryo > --- > > Key: BEAM-1139 > URL: https://issues.apache.org/jira/browse/BEAM-1139 > Project: Beam > Issue Type: Improvement > Components: runner-apex > Reporter: Kenneth Knowles > Assignee: Kenneth Knowles >Priority: Minor > > https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/ > This is not necessarily a bug in the Apex runner, but it looks like this > class cannot be serialized via Kryo while the Apex runner needs it to be. > Probably the fix is to roll-forwards a simple change to make it Kryo > serializable. > It is not clear to me the difference between this test run and others. > Clearly there is a coverage gap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (BEAM-1139) Investigate failures in precommit - Apex & Kryo
[ https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-1139: - Assignee: Kenneth Knowles (was: Thomas Weise) > Investigate failures in precommit - Apex & Kryo > --- > > Key: BEAM-1139 > URL: https://issues.apache.org/jira/browse/BEAM-1139 > Project: Beam > Issue Type: Improvement > Components: runner-apex > Reporter: Kenneth Knowles > Assignee: Kenneth Knowles >Priority: Minor > > https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/ > This is not necessarily a bug in the Apex runner, but it looks like this > class cannot be serialized via Kryo while the Apex runner needs it to be. > Probably the fix is to roll-forwards a simple change to make it Kryo > serializable. > It is not clear to me the difference between this test run and others. > Clearly there is a coverage gap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1089) Jenkins comments on PRs are too many & too large
[ https://issues.apache.org/jira/browse/BEAM-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744080#comment-15744080 ] Kenneth Knowles commented on BEAM-1089: --- [~jasonkuster] I see what you mean [here in our configs|https://github.com/apache/incubator-beam/blob/master/.jenkins/common_job_properties.groovy#L124]. Can the whole block be deleted or some such? > Jenkins comments on PRs are too many & too large > > > Key: BEAM-1089 > URL: https://issues.apache.org/jira/browse/BEAM-1089 > Project: Beam > Issue Type: Bug > Components: testing > Reporter: Kenneth Knowles >Assignee: Jason Kuster > > Lately, I've been finding review comments somewhat drowned out by asfbot > copying build results onto a PR. It also generates a lot of needless email. I > have not yet tried to devise just the right filter, hoping we can just return > to the normal practice of leaving just a commit status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1089) Jenkins comments on PRs are too many & too large
[ https://issues.apache.org/jira/browse/BEAM-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744077#comment-15744077 ] Kenneth Knowles commented on BEAM-1089: --- [~neelesh77] that sounds really great as well! > Jenkins comments on PRs are too many & too large > > > Key: BEAM-1089 > URL: https://issues.apache.org/jira/browse/BEAM-1089 > Project: Beam > Issue Type: Bug > Components: testing > Reporter: Kenneth Knowles >Assignee: Jason Kuster > > Lately, I've been finding review comments somewhat drowned out by asfbot > copying build results onto a PR. It also generates a lot of needless email. I > have not yet tried to devise just the right filter, hoping we can just return > to the normal practice of leaving just a commit status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1143) Timestamps on Jenkins log lines
Kenneth Knowles created BEAM-1143: - Summary: Timestamps on Jenkins log lines Key: BEAM-1143 URL: https://issues.apache.org/jira/browse/BEAM-1143 Project: Beam Issue Type: Improvement Components: testing Reporter: Kenneth Knowles Assignee: Jason Kuster Priority: Minor I suspect this might be doable more universally in the groovy DSL scripts, but we would gain some value by a port of https://github.com/apache/incubator-beam/commit/7f82a573d00a5a30331b7bbb8757e55f4a2d93ae to the most appropriate analog for Jenkins. (in the worst case, just exactly porting the env var) We are currently regularly bottlenecked on build duration/backlog and the time seems to exist outside of the durations accounted for by Maven's usual output. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (BEAM-1139) Investigate failures in precommit - Apex & Kryo
[ https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743997#comment-15743997 ] Kenneth Knowles edited comment on BEAM-1139 at 12/13/16 3:18 AM: - Unfortunately not. Further bummer: the postcommit is not a superset of the precommit so it never broke. That would provide a much easier trail. was (Author: kenn): Unfortunately not. Further bummer: the postcommit does not mirror the precommit so it never broke. That would provide a much easier trail. > Investigate failures in precommit - Apex & Kryo > --- > > Key: BEAM-1139 > URL: https://issues.apache.org/jira/browse/BEAM-1139 > Project: Beam > Issue Type: Improvement > Components: runner-apex > Reporter: Kenneth Knowles >Assignee: Thomas Weise >Priority: Minor > > https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/ > This is not necessarily a bug in the Apex runner, but it looks like this > class cannot be serialized via Kryo while the Apex runner needs it to be. > Probably the fix is to roll-forwards a simple change to make it Kryo > serializable. > It is not clear to me the difference between this test run and others. > Clearly there is a coverage gap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1139) Investigate failures in precommit - Apex & Kryo
[ https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-1139: -- Summary: Investigate failures in precommit - Apex & Kryo (was: Failures in precommit - Apex & Kryo) > Investigate failures in precommit - Apex & Kryo > --- > > Key: BEAM-1139 > URL: https://issues.apache.org/jira/browse/BEAM-1139 > Project: Beam > Issue Type: Improvement > Components: runner-apex >Reporter: Kenneth Knowles >Assignee: Thomas Weise >Priority: Minor > > https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/ > This is not necessarily a bug in the Apex runner, but it looks like this > class cannot be serialized via Kryo while the Apex runner needs it to be. > Probably the fix is to roll-forwards a simple change to make it Kryo > serializable. > It is not clear to me the difference between this test run and others. > Clearly there is a coverage gap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1139) Failures in precommit - Apex & Kryo
[ https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-1139: -- Priority: Minor (was: Blocker) > Failures in precommit - Apex & Kryo > --- > > Key: BEAM-1139 > URL: https://issues.apache.org/jira/browse/BEAM-1139 > Project: Beam > Issue Type: Improvement > Components: runner-apex > Reporter: Kenneth Knowles >Assignee: Thomas Weise >Priority: Minor > > https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/ > This is not necessarily a bug in the Apex runner, but it looks like this > class cannot be serialized via Kryo while the Apex runner needs it to be. > Probably the fix is to roll-forwards a simple change to make it Kryo > serializable. > It is not clear to me the difference between this test run and others. > Clearly there is a coverage gap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1139) Failures in precommit - Apex & Kryo
[ https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743997#comment-15743997 ] Kenneth Knowles commented on BEAM-1139: --- Unfortunately not. Further bummer: the postcommit does not mirror the precommit so it never broke. That would provide a much easier trail. > Failures in precommit - Apex & Kryo > --- > > Key: BEAM-1139 > URL: https://issues.apache.org/jira/browse/BEAM-1139 > Project: Beam > Issue Type: Improvement > Components: runner-apex > Reporter: Kenneth Knowles >Assignee: Thomas Weise >Priority: Blocker > > https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/ > This is not necessarily a bug in the Apex runner, but it looks like this > class cannot be serialized via Kryo while the Apex runner needs it to be. > Probably the fix is to roll-forwards a simple change to make it Kryo > serializable. > It is not clear to me the difference between this test run and others. > Clearly there is a coverage gap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-967) Add Jenkins postcommit for Apex runner RunnableOnService tests
[ https://issues.apache.org/jira/browse/BEAM-967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743877#comment-15743877 ] Kenneth Knowles commented on BEAM-967: -- This would likely have caught BEAM-1139 sooner. Any progress? > Add Jenkins postcommit for Apex runner RunnableOnService tests > -- > > Key: BEAM-967 > URL: https://issues.apache.org/jira/browse/BEAM-967 > Project: Beam > Issue Type: Bug > Components: runner-apex, testing > Reporter: Kenneth Knowles >Assignee: Jason Kuster > > During development of the Apex runner, all RunnableOnService tests were run > in precommit. Now that it has been merged, they are not. It needs the usual > treatment that we give runners. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-843) Use New DoFn Directly in Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743596#comment-15743596 ] Kenneth Knowles commented on BEAM-843: -- Removed the version tag because I think it was just added when it was filed, not deliberately to block release. Aljoscha, feel free to indicate that you really mean to ship this now. We don't mean to remove any intentional tag. > Use New DoFn Directly in Flink Runner > - > > Key: BEAM-843 > URL: https://issues.apache.org/jira/browse/BEAM-843 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Aljoscha Krettek > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-843) Use New DoFn Directly in Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-843: - Fix Version/s: (was: 0.4.0-incubating) > Use New DoFn Directly in Flink Runner > - > > Key: BEAM-843 > URL: https://issues.apache.org/jira/browse/BEAM-843 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Aljoscha Krettek > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-843) Use New DoFn Directly in Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743588#comment-15743588 ] Kenneth Knowles commented on BEAM-843: -- I think the only new capabilities it would allow are not yet implemented anyhow. > Use New DoFn Directly in Flink Runner > - > > Key: BEAM-843 > URL: https://issues.apache.org/jira/browse/BEAM-843 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Aljoscha Krettek > Fix For: 0.4.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1139) Failures in precommit - Apex & Kryo
[ https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-1139: -- Assignee: Thomas Weise (was: Kenneth Knowles) > Failures in precommit - Apex & Kryo > --- > > Key: BEAM-1139 > URL: https://issues.apache.org/jira/browse/BEAM-1139 > Project: Beam > Issue Type: Improvement > Components: runner-apex > Reporter: Kenneth Knowles >Assignee: Thomas Weise >Priority: Blocker > > https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/ > This is not necessarily a bug in the Apex runner, but it looks like this > class cannot be serialized via Kryo while the Apex runner needs it to be. > Probably the fix is to roll-forwards a simple change to make it Kryo > serializable. > It is not clear to me the difference between this test run and others. > Clearly there is a coverage gap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1139) Failures in precommit - Apex & Kryo
Kenneth Knowles created BEAM-1139: - Summary: Failures in precommit - Apex & Kryo Key: BEAM-1139 URL: https://issues.apache.org/jira/browse/BEAM-1139 Project: Beam Issue Type: Improvement Components: runner-apex Reporter: Kenneth Knowles Assignee: Thomas Weise Priority: Blocker https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/ This is not necessarily a bug in the Apex runner, but it looks like this class cannot be serialized via Kryo while the Apex runner needs it to be. Probably the fix is to roll-forwards a simple change to make it Kryo serializable. It is not clear to me the difference between this test run and others. Clearly there is a coverage gap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1138) Consider merging KeyedPValueTrackingVisitor with DirectGraphVisitor
Kenneth Knowles created BEAM-1138: - Summary: Consider merging KeyedPValueTrackingVisitor with DirectGraphVisitor Key: BEAM-1138 URL: https://issues.apache.org/jira/browse/BEAM-1138 Project: Beam Issue Type: Improvement Components: runner-direct Reporter: Kenneth Knowles Assignee: Thomas Groh Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1033) BigQueryMatcher is flaky
[
https://issues.apache.org/jira/browse/BEAM-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-1033:
--
Summary: BigQueryMatcher is flaky (was: WindowedWordCountIT is flaky)
> BigQueryMatcher is flaky
>
>
> Key: BEAM-1033
> URL: https://issues.apache.org/jira/browse/BEAM-1033
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Pei He
>Assignee: Mark Liu
>
> Jenkins link:
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/5145/console
> Running org.apache.beam.examples.WindowedWordCountIT
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 304.282 sec
> <<< FAILURE! - in org.apache.beam.examples.WindowedWordCountIT
> testWindowedWordCountInBatch(org.apache.beam.examples.WindowedWordCountIT)
> Time elapsed: 304.282 sec <<< FAILURE!
> java.lang.AssertionError:
> Expected: Expected checksum is (cd5b52939257e12428a9fa085c32a84dd209b180)
> but: Invalid BigQuery response:
> {"jobComplete":false,"jobReference":{"jobId":"job_0STNX_OD83tQOzo6MvmqXCrk61U","projectId":"apache-beam-testing"},"kind":"bigquery#queryResponse"}
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> at
> org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:164)
> at
> org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:93)
> at
> org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:61)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
> at
> org.apache.beam.examples.WindowedWordCount.main(WindowedWordCount.java:224)
> at
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountPipeline(WindowedWordCountIT.java:88)
> at
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountInBatch(WindowedWordCountIT.java:59)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Results :
> Failed tests:
>
> WindowedWordCountIT.testWindowedWordCountInBatch:59->testWindowedWordCountPipeline:88
>
> Expected: Expected checksum is (cd5b52939257e12428a9fa085c32a84dd209b180)
> but: Invalid BigQuery response:
> {"jobComplete":false,"jobReference":{"jobId":"job_0STNX_OD83tQOzo6MvmqXCrk61U","projectId":"apache-beam-testing"},"kind":"bigquery#queryResponse"}
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-1033) BigQueryMatcher is flaky
[
https://issues.apache.org/jira/browse/BEAM-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742850#comment-15742850
]
Kenneth Knowles commented on BEAM-1033:
---
Changing the title because after BEAM-1135 WindowedWordCountIT does not use
this matcher.
> BigQueryMatcher is flaky
>
>
> Key: BEAM-1033
> URL: https://issues.apache.org/jira/browse/BEAM-1033
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Pei He
>Assignee: Mark Liu
>
> Jenkins link:
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/5145/console
> Running org.apache.beam.examples.WindowedWordCountIT
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 304.282 sec
> <<< FAILURE! - in org.apache.beam.examples.WindowedWordCountIT
> testWindowedWordCountInBatch(org.apache.beam.examples.WindowedWordCountIT)
> Time elapsed: 304.282 sec <<< FAILURE!
> java.lang.AssertionError:
> Expected: Expected checksum is (cd5b52939257e12428a9fa085c32a84dd209b180)
> but: Invalid BigQuery response:
> {"jobComplete":false,"jobReference":{"jobId":"job_0STNX_OD83tQOzo6MvmqXCrk61U","projectId":"apache-beam-testing"},"kind":"bigquery#queryResponse"}
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> at
> org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:164)
> at
> org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:93)
> at
> org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:61)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
> at
> org.apache.beam.examples.WindowedWordCount.main(WindowedWordCount.java:224)
> at
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountPipeline(WindowedWordCountIT.java:88)
> at
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountInBatch(WindowedWordCountIT.java:59)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Results :
> Failed tests:
>
> WindowedWordCountIT.testWindowedWordCountInBatch:59->testWindowedWordCountPipeline:88
>
> Expected: Expected checksum is (cd5b52939257e12428a9fa085c32a84dd209b180)
> but: Invalid BigQuery response:
> {"jobComplete":false,"jobReference":{"jobId":"job_0STNX_OD83tQOzo6MvmqXCrk61U","projectId":"apache-beam-testing"},"kind":"bigquery#queryResponse"}
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (BEAM-1135) Make WindowedWordCount example more independent of runner and execution style
Kenneth Knowles created BEAM-1135: - Summary: Make WindowedWordCount example more independent of runner and execution style Key: BEAM-1135 URL: https://issues.apache.org/jira/browse/BEAM-1135 Project: Beam Issue Type: Improvement Components: examples-java Reporter: Kenneth Knowles Assignee: Kenneth Knowles Fix For: 0.4.0-incubating Right now, WindowedWordCount has not received the same nice revision as our other examples. The PR has been open for a bit I am breaking it out into its own ticket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-35) Remove timer multiplexing and give timers identifiers
[ https://issues.apache.org/jira/browse/BEAM-35?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-35. - Resolution: Fixed Fix Version/s: 0.4.0-incubating > Remove timer multiplexing and give timers identifiers > - > > Key: BEAM-35 > URL: https://issues.apache.org/jira/browse/BEAM-35 > Project: Beam > Issue Type: New Feature > Components: runner-core > Reporter: Kenneth Knowles > Assignee: Kenneth Knowles > Fix For: 0.4.0-incubating > > > Today, timers set by the runner core of the SDK are identified by timestamp, > so multiple timers for the same time result in only one backend timer. This > is runner-specific and obsolete, and makes it unsafe to support timer > deletion because backend timers have no real owner. User-facing timers will > require explicit identifiers anyhow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-1121) Update documentation following rename of PTransform.apply
[ https://issues.apache.org/jira/browse/BEAM-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-1121. --- Resolution: Fixed Fix Version/s: Not applicable > Update documentation following rename of PTransform.apply > - > > Key: BEAM-1121 > URL: https://issues.apache.org/jira/browse/BEAM-1121 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Daniel Halperin > Assignee: Kenneth Knowles > Fix For: Not applicable > > > Since PTransform#apply does not exist any more, significant website > documentation may be wrong. > Fix version: 0.4.0-incubating really just means this needs to be done as part > of the 0.4.0-incubating release, since this change will make it into said > release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (BEAM-1121) Update documentation following rename of PTransform.apply
[ https://issues.apache.org/jira/browse/BEAM-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742785#comment-15742785 ] Kenneth Knowles edited comment on BEAM-1121 at 12/12/16 6:55 PM: - Aside from Javadoc, this was fixed three days ago, by https://github.com/apache/incubator-beam-site/pull/104, tagged with BEAM-438. If you find any omissions, let me know ASAP. was (Author: kenn): Aside from Javadoc, this was fixed three days ago it was filed, by https://github.com/apache/incubator-beam-site/pull/104, tagged with BEAM-438. If you find any omissions, let me know ASAP. > Update documentation following rename of PTransform.apply > - > > Key: BEAM-1121 > URL: https://issues.apache.org/jira/browse/BEAM-1121 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Daniel Halperin > Assignee: Kenneth Knowles > Fix For: Not applicable > > > Since PTransform#apply does not exist any more, significant website > documentation may be wrong. > Fix version: 0.4.0-incubating really just means this needs to be done as part > of the 0.4.0-incubating release, since this change will make it into said > release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (BEAM-1121) Update documentation following rename of PTransform.apply
[ https://issues.apache.org/jira/browse/BEAM-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742785#comment-15742785 ] Kenneth Knowles edited comment on BEAM-1121 at 12/12/16 6:54 PM: - Aside from Javadoc, this was fixed three days ago it was filed, by https://github.com/apache/incubator-beam-site/pull/104, tagged with BEAM-438. If you find any omissions, let me know ASAP. was (Author: kenn): Aside from Javadoc, this was fixed three days before it was filed, by https://github.com/apache/incubator-beam-site/pull/104, tagged with BEAM-438. If you find any omissions, let me know ASAP. > Update documentation following rename of PTransform.apply > - > > Key: BEAM-1121 > URL: https://issues.apache.org/jira/browse/BEAM-1121 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Daniel Halperin > Assignee: Kenneth Knowles > > Since PTransform#apply does not exist any more, significant website > documentation may be wrong. > Fix version: 0.4.0-incubating really just means this needs to be done as part > of the 0.4.0-incubating release, since this change will make it into said > release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1121) Update documentation following rename of PTransform.apply
[ https://issues.apache.org/jira/browse/BEAM-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742785#comment-15742785 ] Kenneth Knowles commented on BEAM-1121: --- Aside from Javadoc, this was fixed three days before it was filed, by https://github.com/apache/incubator-beam-site/pull/104, tagged with BEAM-438. If you find any omissions, let me know ASAP. > Update documentation following rename of PTransform.apply > - > > Key: BEAM-1121 > URL: https://issues.apache.org/jira/browse/BEAM-1121 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Daniel Halperin > Assignee: Kenneth Knowles > > Since PTransform#apply does not exist any more, significant website > documentation may be wrong. > Fix version: 0.4.0-incubating really just means this needs to be done as part > of the 0.4.0-incubating release, since this change will make it into said > release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1132) Jenkins JaCoCo plugin for Beam
Kenneth Knowles created BEAM-1132: - Summary: Jenkins JaCoCo plugin for Beam Key: BEAM-1132 URL: https://issues.apache.org/jira/browse/BEAM-1132 Project: Beam Issue Type: New Feature Components: testing Reporter: Kenneth Knowles Assignee: Davor Bonaci Jenkins has a JaCoCo plugin and other Apache projects use it. We should try it too (and might as well disable coveralls, as I don't know anyone who looks at it). If this takes more than 10 minutes to set up, then another option is to just archive JaCoCo's HTML reports so we can browse them. Either of these should take just minutes and yield huge benefits in visibility of where we have really bad coverage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1131) Consider japicmp for testing backwards-compatibility
Kenneth Knowles created BEAM-1131: - Summary: Consider japicmp for testing backwards-compatibility Key: BEAM-1131 URL: https://issues.apache.org/jira/browse/BEAM-1131 Project: Beam Issue Type: New Feature Components: testing Reporter: Kenneth Knowles Assignee: Jason Kuster japicmp is what Flink uses to test backwards-compatibility, configuration here: https://github.com/apache/flink/blob/41d5875bfc272f2cd5c7e8c8523036684865c1ce/pom.xml#L1184 At our first stable release, we should activate this plugin too! (configuration to be determined by community discussion). We should have it installed and configured beforehand, informational-only, so that we have a sense of it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1115) Support for new Timer API in Spark runner
[ https://issues.apache.org/jira/browse/BEAM-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-1115: -- Component/s: (was: runner-apex) runner-spark > Support for new Timer API in Spark runner > - > > Key: BEAM-1115 > URL: https://issues.apache.org/jira/browse/BEAM-1115 > Project: Beam > Issue Type: New Feature > Components: runner-spark > Reporter: Kenneth Knowles > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (BEAM-1117) Support for new Timer API in Direct runner
[ https://issues.apache.org/jira/browse/BEAM-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-1117: - Assignee: Kenneth Knowles (was: Thomas Groh) > Support for new Timer API in Direct runner > -- > > Key: BEAM-1117 > URL: https://issues.apache.org/jira/browse/BEAM-1117 > Project: Beam > Issue Type: New Feature > Components: runner-direct > Reporter: Kenneth Knowles > Assignee: Kenneth Knowles > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1116) Support for new Timer API in Flink runner
[ https://issues.apache.org/jira/browse/BEAM-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-1116: -- Assignee: (was: Maximilian Michels) > Support for new Timer API in Flink runner > - > > Key: BEAM-1116 > URL: https://issues.apache.org/jira/browse/BEAM-1116 > Project: Beam > Issue Type: New Feature > Components: runner-flink > Reporter: Kenneth Knowles > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1118) Support for new Timer API in Gearpump runner
Kenneth Knowles created BEAM-1118: - Summary: Support for new Timer API in Gearpump runner Key: BEAM-1118 URL: https://issues.apache.org/jira/browse/BEAM-1118 Project: Beam Issue Type: New Feature Components: runner-gearpump Reporter: Kenneth Knowles -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1117) Support for new Timer API in Direct runner
Kenneth Knowles created BEAM-1117: - Summary: Support for new Timer API in Direct runner Key: BEAM-1117 URL: https://issues.apache.org/jira/browse/BEAM-1117 Project: Beam Issue Type: New Feature Components: runner-direct Reporter: Kenneth Knowles Assignee: Thomas Groh -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1116) Support for new Timer API in Flink runner
Kenneth Knowles created BEAM-1116: - Summary: Support for new Timer API in Flink runner Key: BEAM-1116 URL: https://issues.apache.org/jira/browse/BEAM-1116 Project: Beam Issue Type: New Feature Components: runner-flink Reporter: Kenneth Knowles Assignee: Maximilian Michels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1114) Support for new Timer API in Apex runner
Kenneth Knowles created BEAM-1114: - Summary: Support for new Timer API in Apex runner Key: BEAM-1114 URL: https://issues.apache.org/jira/browse/BEAM-1114 Project: Beam Issue Type: New Feature Components: runner-apex Reporter: Kenneth Knowles -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1115) Support for new Timer API in Spark runner
Kenneth Knowles created BEAM-1115: - Summary: Support for new Timer API in Spark runner Key: BEAM-1115 URL: https://issues.apache.org/jira/browse/BEAM-1115 Project: Beam Issue Type: New Feature Components: runner-apex Reporter: Kenneth Knowles -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1113) Support for new Timer API in Dataflow runner
Kenneth Knowles created BEAM-1113: - Summary: Support for new Timer API in Dataflow runner Key: BEAM-1113 URL: https://issues.apache.org/jira/browse/BEAM-1113 Project: Beam Issue Type: New Feature Components: runner-dataflow Reporter: Kenneth Knowles Assignee: Kenneth Knowles -- This message was sent by Atlassian JIRA (v6.3.4#6332)
