Data-dependent file destinations is a pretty great feature. We also have another change to make to this @Experimental feature, and it would be nice to get them both into 2.1.0 if we can unblock this quickly.
I just tried this too, and failed to reproduce it. But Jenkins and Reuven both have a reliable repro. Questionss: - Any ideas about how these configurations differ? - Does this actually affect users? - Once we have another test that catches this issue, can we delete this test? Every other test passes, including the actual example WordCountIT. Since the PR doesn't change primitives, it also seems like it is an existing issue. And the test seems redundant with our other testing but won't get as much maintenance attention. I don't want to stop catching whatever this issue is, though. Kenn On Wed, Jul 5, 2017 at 10:31 AM, Reuven Lax <re...@google.com.invalid> wrote: > Hi Thomas, > > This only happens with https://github.com/apache/beam/pull/3356. > > Reuven > > On Mon, Jul 3, 2017 at 6:11 AM, Thomas Weise <t...@apache.org> wrote: > > > Hi Reuven, > > > > I'm not able to reproduce the issue locally. I was hoping to see which > > thread is attempting to emit the results. In Apex, only the operator > thread > > can emit the results, any other thread that is launched by the operator > > cannot. I'm not aware of ParDo managing separate threads though and > assume > > this must be a race. If you still have the log, can you send it to me? > > > > Thanks, > > Thomas > > > > > > > > On Sat, Jul 1, 2017 at 5:51 AM, Reuven Lax <re...@google.com.invalid> > > wrote: > > > > > pr/3356 fails in the Apex WordCountTest. The failed test is here > > > <https://builds.apache.org/job/beam_PreCommit_Java_ > > > MavenInstall/12829/org.apache.beam$beam-runners-apex/ > > > testReport/org.apache.beam.runners.apex.examples/WordCountTest/ > > > testWordCountExample/> > > > : > > > > > > Upon debugging, it looks like this is likely a problem in the Apex > runner > > > itself. A ParDo calls output(), and that triggers an exception thrown > > from > > > inside the Apex runner. The Apex runner calls emit on a > DefaultOutputPort > > > (ApexParDoOperator.java:275), and that throws an exception inside of > > > verifyOperatorThread(). > > > > > > I'm going to ignore this failure for now as it seems unrelated to my > PR, > > > but does someone want to take a look? > > > > > > Reuven > > > > > >
