I've done a bit more debugging with logging. It appears that the finalize
ParDo is never being invoked in this Apex test (or at least the LOG.info in
that ParDo never runs). This ParDo is run on a constant element (code
snippet below), so it should always run.
PCollection<Void> singletonCollection = p.apply(Create.of((Void) null));
singletonCollection
.apply("Finalize", ParDo.of(new DoFn<Void, Integer>() {
@ProcessElement
public void processElement(ProcessContext c) throws Exception {
LOG.info("Finalizing write operation {}.", writeOperation);
On Wed, Jul 5, 2017 at 11:22 AM, Kenneth Knowles <k...@google.com.invalid>
wrote:
> Data-dependent file destinations is a pretty great feature. We also have
> another change to make to this @Experimental feature, and it would be nice
> to get them both into 2.1.0 if we can unblock this quickly.
>
> I just tried this too, and failed to reproduce it. But Jenkins and Reuven
> both have a reliable repro.
>
> Questionss:
>
> - Any ideas about how these configurations differ?
> - Does this actually affect users?
> - Once we have another test that catches this issue, can we delete this
> test?
>
> Every other test passes, including the actual example WordCountIT. Since
> the PR doesn't change primitives, it also seems like it is an existing
> issue. And the test seems redundant with our other testing but won't get as
> much maintenance attention. I don't want to stop catching whatever this
> issue is, though.
>
> Kenn
>
> On Wed, Jul 5, 2017 at 10:31 AM, Reuven Lax <re...@google.com.invalid>
> wrote:
>
> > Hi Thomas,
> >
> > This only happens with https://github.com/apache/beam/pull/3356.
> >
> > Reuven
> >
> > On Mon, Jul 3, 2017 at 6:11 AM, Thomas Weise <t...@apache.org> wrote:
> >
> > > Hi Reuven,
> > >
> > > I'm not able to reproduce the issue locally. I was hoping to see which
> > > thread is attempting to emit the results. In Apex, only the operator
> > thread
> > > can emit the results, any other thread that is launched by the operator
> > > cannot. I'm not aware of ParDo managing separate threads though and
> > assume
> > > this must be a race. If you still have the log, can you send it to me?
> > >
> > > Thanks,
> > > Thomas
> > >
> > >
> > >
> > > On Sat, Jul 1, 2017 at 5:51 AM, Reuven Lax <re...@google.com.invalid>
> > > wrote:
> > >
> > > > pr/3356 fails in the Apex WordCountTest. The failed test is here
> > > > <https://builds.apache.org/job/beam_PreCommit_Java_
> > > > MavenInstall/12829/org.apache.beam$beam-runners-apex/
> > > > testReport/org.apache.beam.runners.apex.examples/WordCountTest/
> > > > testWordCountExample/>
> > > > :
> > > >
> > > > Upon debugging, it looks like this is likely a problem in the Apex
> > runner
> > > > itself. A ParDo calls output(), and that triggers an exception thrown
> > > from
> > > > inside the Apex runner. The Apex runner calls emit on a
> > DefaultOutputPort
> > > > (ApexParDoOperator.java:275), and that throws an exception inside of
> > > > verifyOperatorThread().
> > > >
> > > > I'm going to ignore this failure for now as it seems unrelated to my
> > PR,
> > > > but does someone want to take a look?
> > > >
> > > > Reuven
> > > >
> > >
> >
>
