If there is no plans to invest in ULR then it makes sense to remove it.
Going forward, however, I think we should try to document the higher level
approach we're taking with runners (and portability) now that we have
something working and can reflect on it. For example, couple of things that
are not 100% clear to me:
- if the focus is on python runner for portability efforts, how does java
SDK (and other languages) tie into this? E.g. how do we run, test, measure,
and develop things (pipelines, aspects of the SDK, runner);
- what's our approach to developing new features, should we make sure
python runner supports them as early as possible (e.g. schemas and SQL)?
- java DirectRunner is still there:
- it is still the primary tool for java SDK development purposes, and
as Kenn mentioned in the linked threads it adds value by making sure users
don't rely on implementation details of specific runners. Do we have a
similar story for portable scenarios?
- I assume that extra validations in the DirectRunner have impact on
performance in various ways (potentially non-deterministic). While this
doesn't matter in some cases, it might do in others. Having a local runner
that is (better) optimized for execution would probably make more sense for
perf measurements, integration tests, and maybe even local production jobs.
Is this something potentially worth looking into?
Regards,
Anton
On Fri, Apr 26, 2019 at 4:41 AM Maximilian Michels <m...@apache.org> wrote:
> Thanks for following up with this. I have mixed feelings to see the
> portable Java DirectRunner go, but I'm in favor of this change because
> it removes a lot of code that we do not really make use of.
>
> -Max
>
> On 26.04.19 02:58, Kenneth Knowles wrote:
> > Thanks for providing all this background on the PR. It is very easy to
> > see where it came from. Definitely nice to have less code and fewer
> > things that can break. Perhaps lazy consensus is enough.
> >
> > Kenn
> >
> > On Thu, Apr 25, 2019 at 4:01 PM Daniel Oliveira <danolive...@google.com
> > <mailto:danolive...@google.com>> wrote:
> >
> > Hey everyone,
> >
> > I made a preliminary PR for removing all the Java Reference Runner
> > code (PR-8380 <https://github.com/apache/beam/pull/8380>) since I
> > wanted to see if it could be done easily. It seems to be working
> > fine, so I wanted to open up this discussion to make sure people are
> > still in agreement on getting rid of this code and that people don't
> > have any concerns.
> >
> > For those who need additional context about this, this previous
> > thread
> > <
> https://lists.apache.org/thread.html/b235f8ee55a737ea399756edd80b1218ed34d3439f7b0ed59bfa8e40@%3Cdev.beam.apache.org%3E
> >
> > is where we discussed deprecating the Java Reference Runner (in some
> > places it's called the ULR or Universal Local Runner, but it's the
> > same thing). Then there's this thread
> > <
> https://lists.apache.org/thread.html/0b68efce9b7f2c5297b32d09e5d903e9b354199fe2ce446fbcd240bc@%3Cdev.beam.apache.org%3E
> >
> > where we discussed removing the code from the repo since it's been
> > deprecated.
> >
> > If no one has any objections to trying to remove the code I'll have
> > someone review the PR I wrote and start a vote to have it merged.
> >
> > Thanks,
> > Daniel Oliveira
> >
>
