+1. I think we should also better reflect connector capabilities (or
include them into features), to avoid surprises like [1].

[1]
https://lists.apache.org/thread.html/9e9270bfb85058e24b762790e948d8bfc558f58ef1df9e14c4e4464c@%3Cuser.beam.apache.org%3E

On Fri, Nov 8, 2019 at 10:51 AM Kenneth Knowles <k...@apache.org> wrote:

>
> On Fri, Nov 8, 2019 at 9:46 AM Brian Hulette <bhule...@google.com> wrote:
>
>> > Does it make sense to do this?
>> I think this makes a lot of sense. Plus it's a good opportunity to
>> refresh the UX of [1].
>>
>
> +1 to total UX refresh. I will advertise
> https://issues.apache.org/jira/browse/BEAM-2888 which has a lot of
> related ideas.
>
>
>> > what's a good way of doing it? Should we expand the existing Capability
>> Matrix to support SDKs as well? Or should we have a new one?
>> To me there are two aspects to this: how we model the data, and how we
>> present the data.
>>
>> For modelling the data:
>> Do we need to maintain the full 3-dimensional <feature - SDK - runner>
>> matrix? That seems untenable to me. With portability, I think the runner
>> and SDK matrix should be completely independent, so it should be safe to
>> just maintain <feature - SDK>, and <feature - runner> matrices and model
>> the 3-dimensional matrix as the cross-product of the two.
>> Maybe we should have a new capability matrix just for portable runners so
>> we can exploit this property?
>>
>
> Agree that we should not do the full product of <# runners> times <#
> SDKs>. That's the whole point of portability.
>
> Early in the project, we deliberately did not include SDKs in the
> capability matrix for philosophical reasons:
>
>  - a runner might support/not support features based on intrinsic
> properties of the underlying engine, not just immaturity
>  - an SDK can have no such intrinsic reason
>
> In practice, though, the matrix is more about maturity than intrinsic
> properties. And SDKs include now a significant runtime component which will
> always take time to implement new model features. I think we should embrace
> the matrix as primarily a measure of maturity and answer Pablo's initial
> question which is most useful for users.
>
> Kenn
>
>
>> For presenting the data:
>> I think there would be value in just presenting <feature - runner>
>> (basically what we have now in [1]), and also presenting <feature - SDK>
>> separately. The <feature - SDK> display could serve as documentation too,
>> with examples of how to do Y in each SDK.
>> Maybe there would also be value in presenting <feature - SDK - runner> in
>> some fancy UI so an architect can quickly answer "what can I do with SDK Z
>> on Runner X", but I'm not sure what that would look like.
>>
>> [1] https://beam.apache.org/documentation/runners/capability-matrix/
>>
>> On Thu, Nov 7, 2019 at 10:09 PM Thomas Weise <t...@apache.org> wrote:
>>
>>> FWIW there are currently at least 2 instances of capability matrix [1]
>>> [2].
>>>
>>> [1] has been in need of a refresh for a while.
>>>
>>> [2] is more useful but only covers portable runners and is hard to find.
>>>
>>> Thomas
>>>
>>> [1] https://beam.apache.org/documentation/runners/capability-matrix/
>>> [2] https://s.apache.org/apache-beam-portability-support-table
>>>
>>> On Thu, Nov 7, 2019 at 7:52 PM Pablo Estrada <pabl...@google.com> wrote:
>>>
>>>> Hi all,
>>>> I think this is a relatively common question:
>>>>
>>>> - Can I do X with runner Y, and SDK Z?
>>>>
>>>> The answers vary significantly between SDK and Runner pairs. This makes
>>>> it such that the current Capability Matrix falls somewhat short when
>>>> potential users / solutions architects / etc are trying to decide to adopt
>>>> Beam, and which Runner / SDK to use.
>>>>
>>>> I think we need to put some effort in building a capability matrix that
>>>> expresses this information - and maintain it updated.
>>>>
>>>> I would like to discuss a few things:
>>>> - Does it make sense to do this?
>>>> - If it does, what's a good way of doing it? Should we expand the
>>>> existing Capability Matrix to support SDKs as well? Or should we have a new
>>>> one?
>>>> - Any other thoughts you may have about the idea.
>>>>
>>>> Best
>>>> -P.
>>>>
>>>

Reply via email to