Maybe add a toggle in flinkoptions to activate it until it is tested and
you are happy with it? Kind of --flinkExperimental=sdf,log,... (random
values). This allows to hit master and continue to work on it.

Le 13 avr. 2018 03:07, "Robert Bradshaw" <> a écrit :

Thomas captures exactly my concerns.

I think we should look at getting everything we can into master (at least
filing JIRAs, the work may take longer) and move what development we can
there. What remains would be a collection of hacks (mostly in the SDKs, but
it's not a feature branch 'cause we'd never want to actually merge it in)
that hopefully we can whittle away (again, JIRAs should be filed for the
items there).
On Thu, Apr 12, 2018 at 5:44 PM Henning Rohde <> wrote:

> + 1 to capture in JIRA what needs to be done.

> The simplest path forward might be to reimplement/cherrypick'n'modify the
changes onto master directly. We would then effectively just abandon the
hacking branch and treat code there as a prototype. Although we would add
components without end2end verification initially, it would allow parallel
progress across the SDKs and the Flink runner. The Go SDK is also already
in master and can help test the migration of the Flink runner changes
before the other SDKs are ready.

> On Thu, Apr 12, 2018 at 4:44 PM Thomas Weise <> wrote:

>> Strong +1 on transitioning all development to the ASF repo.

>> I think a straight move of the hacking branch may still be problematic
though, because it sets the path to continue hacking vs. working towards a
viable milestone that other contributors can base their work off. I would
prefer a state that separates serious development from hacks in a way where
the code does not overlap.

>> Based on Ben's assessment, if most of the hacks are currently are in the
SDK area, then perhaps we can transition everything related to job server
and translation to master so that it is possible to build and work on the
runner there and then only use  the hacked SDKs branch for demos?

>> And maybe discuss an MVP milestone and put together a JIRA view that
shows what remains to be done to get there?

>> Thanks,
>> Thomas

>> On Thu, Apr 12, 2018 at 4:26 PM, Holden Karau <>

>>> So I would be strongly in favour of adding it as a branch on the Apache
repo. This way other folks are more likely to be able to help with the
splitting up and merging process and also while Flink forward is behind us
getting in the practice of doing feature branches on the ASF repo for
collaboration instead of personal github accounts seems like a worthy goal.

>>> On Thu, Apr 12, 2018 at 4:21 PM Robert Bradshaw <>

>>>> I suppose with the hackathon and flink forward behind us, I'm thinking
>>>> should start shifting gears more getting what we have into master in
>>>> production state and less on continuing working on a hacking branch.
If we
>>>> think it'll fairly quick there's no big need to create an official
>>>> and if it's going to be long lived perhaps we should rethink our
>>>> On Thu, Apr 12, 2018 at 3:44 PM Aljoscha Krettek <>
>>>> wrote:

>>>> > I would also be in favour of adding a branch to our main repo. A
>>>> branch on some personal GitHub account can seem a bit sketchy and
adding a
>>>> branch to our repo could make it more visible for people that are
>>>> interested.

>>>> > On 12. Apr 2018, at 15:29, Ben Sidhom <> wrote:

>>>> > I would say that most of it is not suitable for direct merging.
There are
>>>> several reasons for this:

>>>> > Most changes are built on upstream PRs that are either not submitted
>>>> have been rebased before submission.
>>>> > There are some very hacky changes in the Python and Java SDKs to get
>>>> portable pipelines working. For example, hard coding certain options
>>>> baking dependencies into the SDK harness images. These need to be
>>>> implemented correctly in their respective SDKs.
>>>> > Much of the code does not have proper tests and fails simple lint

>>>> > As a concrete example, I tried cherry-picking the changes from
>>>> into master. This is a
>>>> simple change, but there were so many merge conflicts that in the end
>>>> was easier to just reimplement the changes atop master. More
>>>> most changes will require refactoring before actually going in.

>>>> > On Thu, Apr 12, 2018 at 3:16 PM, Robert Bradshaw <

>>>> wrote:

>>>> >> How much of this is not suitable to merging into master directly
(not as
>>>> >> is, but as separate PRs)?
>>>> >> On Thu, Apr 12, 2018 at 3:10 PM Ben Sidhom <>

>>>> >> > Hey all,

>>>> >> > I've been working on a proof-of-concept portable Flink runner
with some
>>>> >> other Beam contributors. We would like to have a point of reference
>>>> the
>>>> >> rest of the Beam community as we integrate this work into master. It
>>>> >> currently lives under
>>>> >>

>>>> >> > I would suggest pulling this into the main ASF repo under an
>>>> >> appropriately-named branch (flink-portable-hacking?). The name
>>>> >> suggest the intention that this branch is not intended to be pulled
>>>> >> master as-is and that it should rather be used as a reference for

>>>> >> > Thoughts?

>>>> >> > --
>>>> >> > -Ben

>>>> > --
>>>> > -Ben

>>> --
>>> Twitter:

Reply via email to