Re: [DISCUSS] BIP reloaded

[Alex Van Boxel]

So how should we do the voting now? Ask for a vote on BIP-1, but what of
open issues... could people add open issues, or should this be done before
the vote?

 _/
_/ Alex Van Boxel


On Mon, Feb 17, 2020 at 4:29 PM Jan Lukavský  wrote:

> Hi Alex,
>
> +1, I think that the current structure is cool.
>
> Jan
> On 2/16/20 10:07 AM, Alex Van Boxel wrote:
>
> OK, I've reordered and tweaked it a bit based on your suggestions, but I'm
> going to stop here. I'm spending far more time on this than the
> implementation.
>
> I did add an open issues section though that people can add issues too
> that can be discussed and voted on. Those that make sense?
>
>  _/
> _/ Alex Van Boxel
>
>
> On Mon, Feb 10, 2020 at 9:57 AM Jan Lukavský  wrote:
>
>> Hi Alex,
>>
>> because it would be super cool, to create a template from the BIP, I'd
>> suggest a few minor changes:
>>
>>  - can we make motivation, current state, alternatives and implementation
>> the same level headings?
>>
>>  - regarding the ordering - in my point of view it makes sense to first
>> define problem (motivation + current state), then to elaborate on _all_
>> options we have to solve the defined problem and then to make a choice
>> (that would be motivation -> current state -> implementation options ->
>> choice on an option). But I agree that once the section is called
>> 'alternatives' (maybe even 'rejected alternatives') it makes more sense to
>> have it _after_ the choice. But the naming might be just a matter of taste,
>> so this might be sorted out later.
>>
>>  - a small fact note - because the BIP should make people ideally
>> involved in voting process, it should be as explanatory as possible - I'm
>> not feeling to be expert on schemas, so it would help me a little more
>> context and maybe example of the "rejected alternatives", how it would look
>> like, so that one can make a decision even when not being involved with
>> schema on a daily basis. Your explanation is probably well understood by
>> people who are experts in the area, but maybe might somewhat limit the
>> audience.
>>
>> What do you think?
>>
>>  Jan
>> On 2/9/20 9:19 PM, Alex Van Boxel wrote:
>>
>> a = motivation
>> b => *added current state in Beam*
>> c = alternatives
>> d = implementation *(I prefer this to define before the alternatives)*
>> e = *rest of document?*
>>
>>  _/
>> _/ Alex Van Boxel
>>
>>
>> On Sun, Feb 9, 2020 at 7:50 PM Jan Lukavský  wrote:
>>
>>> It's absolutely fine. :-) I think that the scope and quality of your
>>> document suits very well for the first BIP.
>>>
>>> What I would find generally useful is a general structure that would be
>>> something like:
>>>
>>>  a) definition of the problem
>>>
>>>  b) explanation why current Beam options don't fit well for the problem
>>> defined at a)
>>>
>>>  c) ideally exhaustive list of possible solutions
>>>
>>>  d) choose of an option from c) with justification of the choice
>>>
>>>  e) implementation notes specific to the choice in d)
>>>
>>> I find mostly the point d) essential, because that can be used as a base
>>> for vote (that is, if the community agrees that the list of options is
>>> exhaustive and that the chosen solution is the best one possible) for
>>> promoting a BIP from proposed to accepted.
>>>
>>> Does that make sense in your case?
>>>
>>>  Jan
>>> On 2/9/20 7:08 PM, Alex Van Boxel wrote:
>>>
>>> I'm sorry, I stole the number 1 from you. Feel free to give suggestions
>>> on the form, so we can get a good template for further BIPs
>>>
>>>  _/
>>> _/ Alex Van Boxel
>>>
>>>
>>> On Sun, Feb 9, 2020 at 6:43 PM Jan Lukavský  wrote:
>>>
 Hi Alex,

 this is cool! Thanks for pushing this topic forward!

 Jan
 On 2/9/20 6:36 PM, Alex Van Boxel wrote:

 BIP-1 is available here:
 https://cwiki.apache.org/confluence/display/BEAM/%5BBIP-1%5D+Beam+Schema+Options

  _/
 _/ Alex Van Boxel


 On Sat, Feb 1, 2020 at 9:11 PM Kenneth Knowles  wrote:

> Sounds great. If you scrape recent dev@ for proposals that are not
> yet implemented, I think you will find some, and you could ask them to add
> as a BIP if they are still interested.
>
> Kenn
>
> On Sat, Feb 1, 2020 at 1:11 AM Jan Lukavský  wrote:
>
>> Hi Kenn,
>>
>> yes, I can do that. I think that there should be at least one first
>> BIP, I can try to setup one. But (as opposed to my previous proposal) 
>> I'll
>> try to setup a fresh one, not the one of [BEAM-8550], because that one
>> already has a PR and rebasing the PR on master for such a long period 
>> (and
>> it is likely, that final polishing of the BIP process will take several
>> more months) starts to be costly. I have in mind two fresh candidates, so
>> I'll pick one of them. I think that only setuping a cwiki would not start
>> the process, we need a real-life example of a BIP included in that.
>>
>> Does that sound ok?
>>
>>  Jan
>> On 

Re: [DISCUSS] BIP reloaded

[Alex Van Boxel]

OK, I've reordered and tweaked it a bit based on your suggestions, but I'm
going to stop here. I'm spending far more time on this than the
implementation.

I did add an open issues section though that people can add issues too that
can be discussed and voted on. Those that make sense?

 _/
_/ Alex Van Boxel


On Mon, Feb 10, 2020 at 9:57 AM Jan Lukavský  wrote:

> Hi Alex,
>
> because it would be super cool, to create a template from the BIP, I'd
> suggest a few minor changes:
>
>  - can we make motivation, current state, alternatives and implementation
> the same level headings?
>
>  - regarding the ordering - in my point of view it makes sense to first
> define problem (motivation + current state), then to elaborate on _all_
> options we have to solve the defined problem and then to make a choice
> (that would be motivation -> current state -> implementation options ->
> choice on an option). But I agree that once the section is called
> 'alternatives' (maybe even 'rejected alternatives') it makes more sense to
> have it _after_ the choice. But the naming might be just a matter of taste,
> so this might be sorted out later.
>
>  - a small fact note - because the BIP should make people ideally involved
> in voting process, it should be as explanatory as possible - I'm not
> feeling to be expert on schemas, so it would help me a little more context
> and maybe example of the "rejected alternatives", how it would look like,
> so that one can make a decision even when not being involved with schema on
> a daily basis. Your explanation is probably well understood by people who
> are experts in the area, but maybe might somewhat limit the audience.
>
> What do you think?
>
>  Jan
> On 2/9/20 9:19 PM, Alex Van Boxel wrote:
>
> a = motivation
> b => *added current state in Beam*
> c = alternatives
> d = implementation *(I prefer this to define before the alternatives)*
> e = *rest of document?*
>
>  _/
> _/ Alex Van Boxel
>
>
> On Sun, Feb 9, 2020 at 7:50 PM Jan Lukavský  wrote:
>
>> It's absolutely fine. :-) I think that the scope and quality of your
>> document suits very well for the first BIP.
>>
>> What I would find generally useful is a general structure that would be
>> something like:
>>
>>  a) definition of the problem
>>
>>  b) explanation why current Beam options don't fit well for the problem
>> defined at a)
>>
>>  c) ideally exhaustive list of possible solutions
>>
>>  d) choose of an option from c) with justification of the choice
>>
>>  e) implementation notes specific to the choice in d)
>>
>> I find mostly the point d) essential, because that can be used as a base
>> for vote (that is, if the community agrees that the list of options is
>> exhaustive and that the chosen solution is the best one possible) for
>> promoting a BIP from proposed to accepted.
>>
>> Does that make sense in your case?
>>
>>  Jan
>> On 2/9/20 7:08 PM, Alex Van Boxel wrote:
>>
>> I'm sorry, I stole the number 1 from you. Feel free to give suggestions
>> on the form, so we can get a good template for further BIPs
>>
>>  _/
>> _/ Alex Van Boxel
>>
>>
>> On Sun, Feb 9, 2020 at 6:43 PM Jan Lukavský  wrote:
>>
>>> Hi Alex,
>>>
>>> this is cool! Thanks for pushing this topic forward!
>>>
>>> Jan
>>> On 2/9/20 6:36 PM, Alex Van Boxel wrote:
>>>
>>> BIP-1 is available here:
>>> https://cwiki.apache.org/confluence/display/BEAM/%5BBIP-1%5D+Beam+Schema+Options
>>>
>>>  _/
>>> _/ Alex Van Boxel
>>>
>>>
>>> On Sat, Feb 1, 2020 at 9:11 PM Kenneth Knowles  wrote:
>>>
 Sounds great. If you scrape recent dev@ for proposals that are not yet
 implemented, I think you will find some, and you could ask them to add as a
 BIP if they are still interested.

 Kenn

 On Sat, Feb 1, 2020 at 1:11 AM Jan Lukavský  wrote:

> Hi Kenn,
>
> yes, I can do that. I think that there should be at least one first
> BIP, I can try to setup one. But (as opposed to my previous proposal) I'll
> try to setup a fresh one, not the one of [BEAM-8550], because that one
> already has a PR and rebasing the PR on master for such a long period (and
> it is likely, that final polishing of the BIP process will take several
> more months) starts to be costly. I have in mind two fresh candidates, so
> I'll pick one of them. I think that only setuping a cwiki would not start
> the process, we need a real-life example of a BIP included in that.
>
> Does that sound ok?
>
>  Jan
> On 2/1/20 5:55 AM, Kenneth Knowles wrote:
>
> These stages sound like a great starting point to me. Would you be the
> volunteer to set up a cwiki page for BIPs?
>
> Kenn
>
> On Mon, Jan 20, 2020 at 3:30 AM Jan Lukavský  wrote:
>
>> I agree that we can take inspiration from other projects. Besides the
>> organizational part (what should be part of BIP, where to store it, how 
>> to
>> edit it and how to make it available to the whole community) - for which
>> 

Re: [DISCUSS] BIP reloaded

[Kenneth Knowles]

I jumped into these wiki pages and figured out how Airflow did theirs using
the Page Properties table on each BIP [1] and how these automatically
update the index using the Page Properties Report [2]. I would consider
creating BIPs for ongoing efforts to flesh out these table, to establish
the columns that matter for each phase of a BIP.

Kenn

[1]
https://confluence.atlassian.com/conf71/page-properties-macro-979423418.html
[2]
https://confluence.atlassian.com/conf71/page-properties-report-macro-979423430.html

On Mon, Feb 10, 2020 at 12:57 AM Jan Lukavský  wrote:

> Hi Alex,
>
> because it would be super cool, to create a template from the BIP, I'd
> suggest a few minor changes:
>
>  - can we make motivation, current state, alternatives and implementation
> the same level headings?
>
>  - regarding the ordering - in my point of view it makes sense to first
> define problem (motivation + current state), then to elaborate on _all_
> options we have to solve the defined problem and then to make a choice
> (that would be motivation -> current state -> implementation options ->
> choice on an option). But I agree that once the section is called
> 'alternatives' (maybe even 'rejected alternatives') it makes more sense to
> have it _after_ the choice. But the naming might be just a matter of taste,
> so this might be sorted out later.
>
>  - a small fact note - because the BIP should make people ideally involved
> in voting process, it should be as explanatory as possible - I'm not
> feeling to be expert on schemas, so it would help me a little more context
> and maybe example of the "rejected alternatives", how it would look like,
> so that one can make a decision even when not being involved with schema on
> a daily basis. Your explanation is probably well understood by people who
> are experts in the area, but maybe might somewhat limit the audience.
>
> What do you think?
>
>  Jan
> On 2/9/20 9:19 PM, Alex Van Boxel wrote:
>
> a = motivation
> b => *added current state in Beam*
> c = alternatives
> d = implementation *(I prefer this to define before the alternatives)*
> e = *rest of document?*
>
>  _/
> _/ Alex Van Boxel
>
>
> On Sun, Feb 9, 2020 at 7:50 PM Jan Lukavský  wrote:
>
>> It's absolutely fine. :-) I think that the scope and quality of your
>> document suits very well for the first BIP.
>>
>> What I would find generally useful is a general structure that would be
>> something like:
>>
>>  a) definition of the problem
>>
>>  b) explanation why current Beam options don't fit well for the problem
>> defined at a)
>>
>>  c) ideally exhaustive list of possible solutions
>>
>>  d) choose of an option from c) with justification of the choice
>>
>>  e) implementation notes specific to the choice in d)
>>
>> I find mostly the point d) essential, because that can be used as a base
>> for vote (that is, if the community agrees that the list of options is
>> exhaustive and that the chosen solution is the best one possible) for
>> promoting a BIP from proposed to accepted.
>>
>> Does that make sense in your case?
>>
>>  Jan
>> On 2/9/20 7:08 PM, Alex Van Boxel wrote:
>>
>> I'm sorry, I stole the number 1 from you. Feel free to give suggestions
>> on the form, so we can get a good template for further BIPs
>>
>>  _/
>> _/ Alex Van Boxel
>>
>>
>> On Sun, Feb 9, 2020 at 6:43 PM Jan Lukavský  wrote:
>>
>>> Hi Alex,
>>>
>>> this is cool! Thanks for pushing this topic forward!
>>>
>>> Jan
>>> On 2/9/20 6:36 PM, Alex Van Boxel wrote:
>>>
>>> BIP-1 is available here:
>>> https://cwiki.apache.org/confluence/display/BEAM/%5BBIP-1%5D+Beam+Schema+Options
>>>
>>>  _/
>>> _/ Alex Van Boxel
>>>
>>>
>>> On Sat, Feb 1, 2020 at 9:11 PM Kenneth Knowles  wrote:
>>>
 Sounds great. If you scrape recent dev@ for proposals that are not yet
 implemented, I think you will find some, and you could ask them to add as a
 BIP if they are still interested.

 Kenn

 On Sat, Feb 1, 2020 at 1:11 AM Jan Lukavský  wrote:

> Hi Kenn,
>
> yes, I can do that. I think that there should be at least one first
> BIP, I can try to setup one. But (as opposed to my previous proposal) I'll
> try to setup a fresh one, not the one of [BEAM-8550], because that one
> already has a PR and rebasing the PR on master for such a long period (and
> it is likely, that final polishing of the BIP process will take several
> more months) starts to be costly. I have in mind two fresh candidates, so
> I'll pick one of them. I think that only setuping a cwiki would not start
> the process, we need a real-life example of a BIP included in that.
>
> Does that sound ok?
>
>  Jan
> On 2/1/20 5:55 AM, Kenneth Knowles wrote:
>
> These stages sound like a great starting point to me. Would you be the
> volunteer to set up a cwiki page for BIPs?
>
> Kenn
>
> On Mon, Jan 20, 2020 at 3:30 AM Jan Lukavský  wrote:
>
>> I agree that we can take inspiration from 

Re: [DISCUSS] BIP reloaded

[Alex Van Boxel]

a = motivation
b => *added current state in Beam*
c = alternatives
d = implementation *(I prefer this to define before the alternatives)*
e = *rest of document?*

 _/
_/ Alex Van Boxel


On Sun, Feb 9, 2020 at 7:50 PM Jan Lukavský  wrote:

> It's absolutely fine. :-) I think that the scope and quality of your
> document suits very well for the first BIP.
>
> What I would find generally useful is a general structure that would be
> something like:
>
>  a) definition of the problem
>
>  b) explanation why current Beam options don't fit well for the problem
> defined at a)
>
>  c) ideally exhaustive list of possible solutions
>
>  d) choose of an option from c) with justification of the choice
>
>  e) implementation notes specific to the choice in d)
>
> I find mostly the point d) essential, because that can be used as a base
> for vote (that is, if the community agrees that the list of options is
> exhaustive and that the chosen solution is the best one possible) for
> promoting a BIP from proposed to accepted.
>
> Does that make sense in your case?
>
>  Jan
> On 2/9/20 7:08 PM, Alex Van Boxel wrote:
>
> I'm sorry, I stole the number 1 from you. Feel free to give suggestions on
> the form, so we can get a good template for further BIPs
>
>  _/
> _/ Alex Van Boxel
>
>
> On Sun, Feb 9, 2020 at 6:43 PM Jan Lukavský  wrote:
>
>> Hi Alex,
>>
>> this is cool! Thanks for pushing this topic forward!
>>
>> Jan
>> On 2/9/20 6:36 PM, Alex Van Boxel wrote:
>>
>> BIP-1 is available here:
>> https://cwiki.apache.org/confluence/display/BEAM/%5BBIP-1%5D+Beam+Schema+Options
>>
>>  _/
>> _/ Alex Van Boxel
>>
>>
>> On Sat, Feb 1, 2020 at 9:11 PM Kenneth Knowles  wrote:
>>
>>> Sounds great. If you scrape recent dev@ for proposals that are not yet
>>> implemented, I think you will find some, and you could ask them to add as a
>>> BIP if they are still interested.
>>>
>>> Kenn
>>>
>>> On Sat, Feb 1, 2020 at 1:11 AM Jan Lukavský  wrote:
>>>
 Hi Kenn,

 yes, I can do that. I think that there should be at least one first
 BIP, I can try to setup one. But (as opposed to my previous proposal) I'll
 try to setup a fresh one, not the one of [BEAM-8550], because that one
 already has a PR and rebasing the PR on master for such a long period (and
 it is likely, that final polishing of the BIP process will take several
 more months) starts to be costly. I have in mind two fresh candidates, so
 I'll pick one of them. I think that only setuping a cwiki would not start
 the process, we need a real-life example of a BIP included in that.

 Does that sound ok?

  Jan
 On 2/1/20 5:55 AM, Kenneth Knowles wrote:

 These stages sound like a great starting point to me. Would you be the
 volunteer to set up a cwiki page for BIPs?

 Kenn

 On Mon, Jan 20, 2020 at 3:30 AM Jan Lukavský  wrote:

> I agree that we can take inspiration from other projects. Besides the
> organizational part (what should be part of BIP, where to store it, how to
> edit it and how to make it available to the whole community) - for which
> the cwiki might be the best option - there are still open questions about
> the lifecycle of a BIP:
>
>  a) when to create one?
>
>   - I feel this might be optional, there might be some upper bound of
> features that are really "too big" or "too controversial", so these should
> undergo the BIP process in all cases, otherwise the decision might be part
> of the proposal, the question is how to define those
>
>  b) what are lifecycle stages of a BIP? How to do transitions between
> these stages?
>
>   - From the top of my head this might be:
>
> a) proposal -- not yet accepted
>
> b) accepted -- most probably after vote?
>
> c) in progress -- having assigned JIRA and being worked on
>
> d) done -- after merge to master
>
> e) released -- obvious
>
> WDYT?
>
>  Jan
> On 1/15/20 8:19 PM, Kenneth Knowles wrote:
>
> Focusing this thread on the BIP process seems wise, without changing
> much else in the same thread. I don't think the BIP process has to do with
> exactly how design docs are written or archived, but the ability to *at a
> glance* understand:
>
>  - what are the high level proposals
>  - status of the proposals
>  - who to contact
>  - how to get to more info (links to design docs, thread, Jiras, etc)
>
> A page with a table on cwiki is common and seems good for this. How we
> manage such a table would be a possible next step. I think they should
> focus on large changes that need heavyweight process, so should encourage
> lightweight creation. I think adding heavy process to smaller changes 
> would
> be bad.
>
> 
>
> I have looked multiple times at other projects (linked in prior thread
> and in this 

Re: [DISCUSS] BIP reloaded

[Jan Lukavský]

It's absolutely fine. :-) I think that the scope and quality of your 
document suits very well for the first BIP.


What I would find generally useful is a general structure that would be 
something like:


 a) definition of the problem

 b) explanation why current Beam options don't fit well for the problem 
defined at a)


 c) ideally exhaustive list of possible solutions

 d) choose of an option from c) with justification of the choice

 e) implementation notes specific to the choice in d)

I find mostly the point d) essential, because that can be used as a base 
for vote (that is, if the community agrees that the list of options is 
exhaustive and that the chosen solution is the best one possible) for 
promoting a BIP from proposed to accepted.


Does that make sense in your case?

 Jan

On 2/9/20 7:08 PM, Alex Van Boxel wrote:
I'm sorry, I stole the number 1 from you. Feel free to give 
suggestions on the form, so we can get a good template for further BIPs


 _/
_/ Alex Van Boxel


On Sun, Feb 9, 2020 at 6:43 PM Jan Lukavský > wrote:


Hi Alex,

this is cool! Thanks for pushing this topic forward!

Jan

On 2/9/20 6:36 PM, Alex Van Boxel wrote:

BIP-1 is available here:

https://cwiki.apache.org/confluence/display/BEAM/%5BBIP-1%5D+Beam+Schema+Options


 _/
_/ Alex Van Boxel


On Sat, Feb 1, 2020 at 9:11 PM Kenneth Knowles mailto:k...@apache.org>> wrote:

Sounds great. If you scrape recent dev@ for proposals that
are not yet implemented, I think you will find some, and you
could ask them to add as a BIP if they are still interested.

Kenn

On Sat, Feb 1, 2020 at 1:11 AM Jan Lukavský mailto:je...@seznam.cz>> wrote:

Hi Kenn,

yes, I can do that. I think that there should be at least
one first BIP, I can try to setup one. But (as opposed to
my previous proposal) I'll try to setup a fresh one, not
the one of [BEAM-8550], because that one already has a PR
and rebasing the PR on master for such a long period (and
it is likely, that final polishing of the BIP process
will take several more months) starts to be costly. I
have in mind two fresh candidates, so I'll pick one of
them. I think that only setuping a cwiki would not start
the process, we need a real-life example of a BIP
included in that.

Does that sound ok?

 Jan

On 2/1/20 5:55 AM, Kenneth Knowles wrote:

These stages sound like a great starting point to me.
Would you be the volunteer to set up a cwiki page for BIPs?

Kenn

On Mon, Jan 20, 2020 at 3:30 AM Jan Lukavský
mailto:je...@seznam.cz>> wrote:

I agree that we can take inspiration from other
projects. Besides the organizational part (what
should be part of BIP, where to store it, how to
edit it and how to make it available to the whole
community) - for which the cwiki might be the best
option - there are still open questions about the
lifecycle of a BIP:

 a) when to create one?

  - I feel this might be optional, there might be
some upper bound of features that are really "too
big" or "too controversial", so these should undergo
the BIP process in all cases, otherwise the decision
might be part of the proposal, the question is how
to define those

 b) what are lifecycle stages of a BIP? How to do
transitions between these stages?

  - From the top of my head this might be:

    a) proposal -- not yet accepted

    b) accepted -- most probably after vote?

    c) in progress -- having assigned JIRA and being
worked on

    d) done -- after merge to master

    e) released -- obvious

WDYT?

 Jan

On 1/15/20 8:19 PM, Kenneth Knowles wrote:

Focusing this thread on the BIP process seems wise,
without changing much else in the same thread. I
don't think the BIP process has to do with exactly
how design docs are written or archived, but the
ability to *at a glance* understand:

 - what are the high level proposals
 - status of the proposals
 - who to contact
 - how to get to more info (links to design docs,
thread, Jiras, etc)

A page with a table on cwiki is common and seems
good for this. How we manage such a table would be
a possible next step. I 

Re: [DISCUSS] BIP reloaded

[Alex Van Boxel]

I'm sorry, I stole the number 1 from you. Feel free to give suggestions on
the form, so we can get a good template for further BIPs

 _/
_/ Alex Van Boxel


On Sun, Feb 9, 2020 at 6:43 PM Jan Lukavský  wrote:

> Hi Alex,
>
> this is cool! Thanks for pushing this topic forward!
>
> Jan
> On 2/9/20 6:36 PM, Alex Van Boxel wrote:
>
> BIP-1 is available here:
> https://cwiki.apache.org/confluence/display/BEAM/%5BBIP-1%5D+Beam+Schema+Options
>
>  _/
> _/ Alex Van Boxel
>
>
> On Sat, Feb 1, 2020 at 9:11 PM Kenneth Knowles  wrote:
>
>> Sounds great. If you scrape recent dev@ for proposals that are not yet
>> implemented, I think you will find some, and you could ask them to add as a
>> BIP if they are still interested.
>>
>> Kenn
>>
>> On Sat, Feb 1, 2020 at 1:11 AM Jan Lukavský  wrote:
>>
>>> Hi Kenn,
>>>
>>> yes, I can do that. I think that there should be at least one first BIP,
>>> I can try to setup one. But (as opposed to my previous proposal) I'll try
>>> to setup a fresh one, not the one of [BEAM-8550], because that one already
>>> has a PR and rebasing the PR on master for such a long period (and it is
>>> likely, that final polishing of the BIP process will take several more
>>> months) starts to be costly. I have in mind two fresh candidates, so I'll
>>> pick one of them. I think that only setuping a cwiki would not start the
>>> process, we need a real-life example of a BIP included in that.
>>>
>>> Does that sound ok?
>>>
>>>  Jan
>>> On 2/1/20 5:55 AM, Kenneth Knowles wrote:
>>>
>>> These stages sound like a great starting point to me. Would you be the
>>> volunteer to set up a cwiki page for BIPs?
>>>
>>> Kenn
>>>
>>> On Mon, Jan 20, 2020 at 3:30 AM Jan Lukavský  wrote:
>>>
 I agree that we can take inspiration from other projects. Besides the
 organizational part (what should be part of BIP, where to store it, how to
 edit it and how to make it available to the whole community) - for which
 the cwiki might be the best option - there are still open questions about
 the lifecycle of a BIP:

  a) when to create one?

   - I feel this might be optional, there might be some upper bound of
 features that are really "too big" or "too controversial", so these should
 undergo the BIP process in all cases, otherwise the decision might be part
 of the proposal, the question is how to define those

  b) what are lifecycle stages of a BIP? How to do transitions between
 these stages?

   - From the top of my head this might be:

 a) proposal -- not yet accepted

 b) accepted -- most probably after vote?

 c) in progress -- having assigned JIRA and being worked on

 d) done -- after merge to master

 e) released -- obvious

 WDYT?

  Jan
 On 1/15/20 8:19 PM, Kenneth Knowles wrote:

 Focusing this thread on the BIP process seems wise, without changing
 much else in the same thread. I don't think the BIP process has to do with
 exactly how design docs are written or archived, but the ability to *at a
 glance* understand:

  - what are the high level proposals
  - status of the proposals
  - who to contact
  - how to get to more info (links to design docs, thread, Jiras, etc)

 A page with a table on cwiki is common and seems good for this. How we
 manage such a table would be a possible next step. I think they should
 focus on large changes that need heavyweight process, so should encourage
 lightweight creation. I think adding heavy process to smaller changes would
 be bad.

 

 I have looked multiple times at other projects (linked in prior thread
 and in this thread too but gathering them here)

 Spark: https://spark.apache.org/improvement-proposals.html
  - Jira is not good for "at a glance" reading. The title should have a
 short and easy to understand paragraph.

 Kafka:
 https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
  - Quite a lot of content; I would prefer 10s of proposals. But it is
 readable enough. Table lacks important content like links and summaries.
  - Blends the table with a bunch of header material that IMO ets in the
 way

 Flink:
 https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
  - Looks very similar to Kafka
  - Target Release is too specific, and actual status is missing

 Airflow:
 https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals
  - seems best organized, and the table has more info
  - having sections for the different status proposals in different
 tables is great
  - "InRelease" column is left blank

 It seems there is a lot of redundancy with Jira fields - owner,
 release, etc. I think that redundancy is good. If it is too much effort to
 redundantly 

Re: [DISCUSS] BIP reloaded

[Jan Lukavský]

Hi Alex,

this is cool! Thanks for pushing this topic forward!

Jan

On 2/9/20 6:36 PM, Alex Van Boxel wrote:
BIP-1 is available here: 
https://cwiki.apache.org/confluence/display/BEAM/%5BBIP-1%5D+Beam+Schema+Options 



 _/
_/ Alex Van Boxel


On Sat, Feb 1, 2020 at 9:11 PM Kenneth Knowles > wrote:


Sounds great. If you scrape recent dev@ for proposals that are not
yet implemented, I think you will find some, and you could ask
them to add as a BIP if they are still interested.

Kenn

On Sat, Feb 1, 2020 at 1:11 AM Jan Lukavský mailto:je...@seznam.cz>> wrote:

Hi Kenn,

yes, I can do that. I think that there should be at least one
first BIP, I can try to setup one. But (as opposed to my
previous proposal) I'll try to setup a fresh one, not the one
of [BEAM-8550], because that one already has a PR and rebasing
the PR on master for such a long period (and it is likely,
that final polishing of the BIP process will take several more
months) starts to be costly. I have in mind two fresh
candidates, so I'll pick one of them. I think that only
setuping a cwiki would not start the process, we need a
real-life example of a BIP included in that.

Does that sound ok?

 Jan

On 2/1/20 5:55 AM, Kenneth Knowles wrote:

These stages sound like a great starting point to me. Would
you be the volunteer to set up a cwiki page for BIPs?

Kenn

On Mon, Jan 20, 2020 at 3:30 AM Jan Lukavský mailto:je...@seznam.cz>> wrote:

I agree that we can take inspiration from other projects.
Besides the organizational part (what should be part of
BIP, where to store it, how to edit it and how to make it
available to the whole community) - for which the cwiki
might be the best option - there are still open questions
about the lifecycle of a BIP:

 a) when to create one?

  - I feel this might be optional, there might be some
upper bound of features that are really "too big" or "too
controversial", so these should undergo the BIP process
in all cases, otherwise the decision might be part of the
proposal, the question is how to define those

 b) what are lifecycle stages of a BIP? How to do
transitions between these stages?

  - From the top of my head this might be:

    a) proposal -- not yet accepted

    b) accepted -- most probably after vote?

    c) in progress -- having assigned JIRA and being
worked on

    d) done -- after merge to master

    e) released -- obvious

WDYT?

 Jan

On 1/15/20 8:19 PM, Kenneth Knowles wrote:

Focusing this thread on the BIP process seems wise,
without changing much else in the same thread. I don't
think the BIP process has to do with exactly how design
docs are written or archived, but the ability to *at a
glance* understand:

 - what are the high level proposals
 - status of the proposals
 - who to contact
 - how to get to more info (links to design docs,
thread, Jiras, etc)

A page with a table on cwiki is common and seems good
for this. How we manage such a table would be a possible
next step. I think they should focus on large changes
that need heavyweight process, so should encourage
lightweight creation. I think adding heavy process to
smaller changes would be bad.



I have looked multiple times at other projects (linked
in prior thread and in this thread too but gathering
them here)

Spark: https://spark.apache.org/improvement-proposals.html
 - Jira is not good for "at a glance" reading. The title
should have a short and easy to understand paragraph.

Kafka:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
 - Quite a lot of content; I would prefer 10s of
proposals. But it is readable enough. Table lacks
important content like links and summaries.
 - Blends the table with a bunch of header material that
IMO ets in the way

Flink:

https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
 - Looks very similar to Kafka
 - Target Release is too specific, and actual status is
missing

Airflow:

https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals
 - seems best organized, and the table has more info
  

Re: [DISCUSS] BIP reloaded

[Alex Van Boxel]

BIP-1 is available here:
https://cwiki.apache.org/confluence/display/BEAM/%5BBIP-1%5D+Beam+Schema+Options

 _/
_/ Alex Van Boxel


On Sat, Feb 1, 2020 at 9:11 PM Kenneth Knowles  wrote:

> Sounds great. If you scrape recent dev@ for proposals that are not yet
> implemented, I think you will find some, and you could ask them to add as a
> BIP if they are still interested.
>
> Kenn
>
> On Sat, Feb 1, 2020 at 1:11 AM Jan Lukavský  wrote:
>
>> Hi Kenn,
>>
>> yes, I can do that. I think that there should be at least one first BIP,
>> I can try to setup one. But (as opposed to my previous proposal) I'll try
>> to setup a fresh one, not the one of [BEAM-8550], because that one already
>> has a PR and rebasing the PR on master for such a long period (and it is
>> likely, that final polishing of the BIP process will take several more
>> months) starts to be costly. I have in mind two fresh candidates, so I'll
>> pick one of them. I think that only setuping a cwiki would not start the
>> process, we need a real-life example of a BIP included in that.
>>
>> Does that sound ok?
>>
>>  Jan
>> On 2/1/20 5:55 AM, Kenneth Knowles wrote:
>>
>> These stages sound like a great starting point to me. Would you be the
>> volunteer to set up a cwiki page for BIPs?
>>
>> Kenn
>>
>> On Mon, Jan 20, 2020 at 3:30 AM Jan Lukavský  wrote:
>>
>>> I agree that we can take inspiration from other projects. Besides the
>>> organizational part (what should be part of BIP, where to store it, how to
>>> edit it and how to make it available to the whole community) - for which
>>> the cwiki might be the best option - there are still open questions about
>>> the lifecycle of a BIP:
>>>
>>>  a) when to create one?
>>>
>>>   - I feel this might be optional, there might be some upper bound of
>>> features that are really "too big" or "too controversial", so these should
>>> undergo the BIP process in all cases, otherwise the decision might be part
>>> of the proposal, the question is how to define those
>>>
>>>  b) what are lifecycle stages of a BIP? How to do transitions between
>>> these stages?
>>>
>>>   - From the top of my head this might be:
>>>
>>> a) proposal -- not yet accepted
>>>
>>> b) accepted -- most probably after vote?
>>>
>>> c) in progress -- having assigned JIRA and being worked on
>>>
>>> d) done -- after merge to master
>>>
>>> e) released -- obvious
>>>
>>> WDYT?
>>>
>>>  Jan
>>> On 1/15/20 8:19 PM, Kenneth Knowles wrote:
>>>
>>> Focusing this thread on the BIP process seems wise, without changing
>>> much else in the same thread. I don't think the BIP process has to do with
>>> exactly how design docs are written or archived, but the ability to *at a
>>> glance* understand:
>>>
>>>  - what are the high level proposals
>>>  - status of the proposals
>>>  - who to contact
>>>  - how to get to more info (links to design docs, thread, Jiras, etc)
>>>
>>> A page with a table on cwiki is common and seems good for this. How we
>>> manage such a table would be a possible next step. I think they should
>>> focus on large changes that need heavyweight process, so should encourage
>>> lightweight creation. I think adding heavy process to smaller changes would
>>> be bad.
>>>
>>> 
>>>
>>> I have looked multiple times at other projects (linked in prior thread
>>> and in this thread too but gathering them here)
>>>
>>> Spark: https://spark.apache.org/improvement-proposals.html
>>>  - Jira is not good for "at a glance" reading. The title should have a
>>> short and easy to understand paragraph.
>>>
>>> Kafka:
>>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>>>  - Quite a lot of content; I would prefer 10s of proposals. But it is
>>> readable enough. Table lacks important content like links and summaries.
>>>  - Blends the table with a bunch of header material that IMO ets in the
>>> way
>>>
>>> Flink:
>>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>>>  - Looks very similar to Kafka
>>>  - Target Release is too specific, and actual status is missing
>>>
>>> Airflow:
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals
>>>  - seems best organized, and the table has more info
>>>  - having sections for the different status proposals in different
>>> tables is great
>>>  - "InRelease" column is left blank
>>>
>>> It seems there is a lot of redundancy with Jira fields - owner, release,
>>> etc. I think that redundancy is good. If it is too much effort to
>>> redundantly manage to write it in the table then it probably is not
>>> appropriate for heavyweight process. Anything that is one simple task that
>>> fits in a Jira that can be passed around from person to person shouldn't be
>>> a BIP. Probably anything where we can guess the target version isn't big
>>> enough for a BIP.
>>>
>>> Kenn
>>>
>>> On Thu, Jan 9, 2020 at 7:59 AM Jan Lukavský  wrote:
>>>
 I think that, besides ownership of a feature, a BIP (or 

Re: [DISCUSS] BIP reloaded

[Kenneth Knowles]

Sounds great. If you scrape recent dev@ for proposals that are not yet
implemented, I think you will find some, and you could ask them to add as a
BIP if they are still interested.

Kenn

On Sat, Feb 1, 2020 at 1:11 AM Jan Lukavský  wrote:

> Hi Kenn,
>
> yes, I can do that. I think that there should be at least one first BIP, I
> can try to setup one. But (as opposed to my previous proposal) I'll try to
> setup a fresh one, not the one of [BEAM-8550], because that one already has
> a PR and rebasing the PR on master for such a long period (and it is
> likely, that final polishing of the BIP process will take several more
> months) starts to be costly. I have in mind two fresh candidates, so I'll
> pick one of them. I think that only setuping a cwiki would not start the
> process, we need a real-life example of a BIP included in that.
>
> Does that sound ok?
>
>  Jan
> On 2/1/20 5:55 AM, Kenneth Knowles wrote:
>
> These stages sound like a great starting point to me. Would you be the
> volunteer to set up a cwiki page for BIPs?
>
> Kenn
>
> On Mon, Jan 20, 2020 at 3:30 AM Jan Lukavský  wrote:
>
>> I agree that we can take inspiration from other projects. Besides the
>> organizational part (what should be part of BIP, where to store it, how to
>> edit it and how to make it available to the whole community) - for which
>> the cwiki might be the best option - there are still open questions about
>> the lifecycle of a BIP:
>>
>>  a) when to create one?
>>
>>   - I feel this might be optional, there might be some upper bound of
>> features that are really "too big" or "too controversial", so these should
>> undergo the BIP process in all cases, otherwise the decision might be part
>> of the proposal, the question is how to define those
>>
>>  b) what are lifecycle stages of a BIP? How to do transitions between
>> these stages?
>>
>>   - From the top of my head this might be:
>>
>> a) proposal -- not yet accepted
>>
>> b) accepted -- most probably after vote?
>>
>> c) in progress -- having assigned JIRA and being worked on
>>
>> d) done -- after merge to master
>>
>> e) released -- obvious
>>
>> WDYT?
>>
>>  Jan
>> On 1/15/20 8:19 PM, Kenneth Knowles wrote:
>>
>> Focusing this thread on the BIP process seems wise, without changing much
>> else in the same thread. I don't think the BIP process has to do with
>> exactly how design docs are written or archived, but the ability to *at a
>> glance* understand:
>>
>>  - what are the high level proposals
>>  - status of the proposals
>>  - who to contact
>>  - how to get to more info (links to design docs, thread, Jiras, etc)
>>
>> A page with a table on cwiki is common and seems good for this. How we
>> manage such a table would be a possible next step. I think they should
>> focus on large changes that need heavyweight process, so should encourage
>> lightweight creation. I think adding heavy process to smaller changes would
>> be bad.
>>
>> 
>>
>> I have looked multiple times at other projects (linked in prior thread
>> and in this thread too but gathering them here)
>>
>> Spark: https://spark.apache.org/improvement-proposals.html
>>  - Jira is not good for "at a glance" reading. The title should have a
>> short and easy to understand paragraph.
>>
>> Kafka:
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>>  - Quite a lot of content; I would prefer 10s of proposals. But it is
>> readable enough. Table lacks important content like links and summaries.
>>  - Blends the table with a bunch of header material that IMO ets in the
>> way
>>
>> Flink:
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>>  - Looks very similar to Kafka
>>  - Target Release is too specific, and actual status is missing
>>
>> Airflow:
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals
>>  - seems best organized, and the table has more info
>>  - having sections for the different status proposals in different tables
>> is great
>>  - "InRelease" column is left blank
>>
>> It seems there is a lot of redundancy with Jira fields - owner, release,
>> etc. I think that redundancy is good. If it is too much effort to
>> redundantly manage to write it in the table then it probably is not
>> appropriate for heavyweight process. Anything that is one simple task that
>> fits in a Jira that can be passed around from person to person shouldn't be
>> a BIP. Probably anything where we can guess the target version isn't big
>> enough for a BIP.
>>
>> Kenn
>>
>> On Thu, Jan 9, 2020 at 7:59 AM Jan Lukavský  wrote:
>>
>>> I think that, besides ownership of a feature, a BIP (or whatever
>>> document or process) should contain the following:
>>>
>>>  * description of a problem that the improvement addresses  - this is
>>> currently often part of design doc
>>>
>>>  * description of multiple possible solutions (if multiple exist, which
>>> is probably mostly the case)
>>>
>>>  * justifying 

Re: [DISCUSS] BIP reloaded

[Jan Lukavský]

Hi Kenn,

yes, I can do that. I think that there should be at least one first BIP, 
I can try to setup one. But (as opposed to my previous proposal) I'll 
try to setup a fresh one, not the one of [BEAM-8550], because that one 
already has a PR and rebasing the PR on master for such a long period 
(and it is likely, that final polishing of the BIP process will take 
several more months) starts to be costly. I have in mind two fresh 
candidates, so I'll pick one of them. I think that only setuping a cwiki 
would not start the process, we need a real-life example of a BIP 
included in that.


Does that sound ok?

 Jan

On 2/1/20 5:55 AM, Kenneth Knowles wrote:
These stages sound like a great starting point to me. Would you be the 
volunteer to set up a cwiki page for BIPs?


Kenn

On Mon, Jan 20, 2020 at 3:30 AM Jan Lukavský > wrote:


I agree that we can take inspiration from other projects. Besides
the organizational part (what should be part of BIP, where to
store it, how to edit it and how to make it available to the whole
community) - for which the cwiki might be the best option - there
are still open questions about the lifecycle of a BIP:

 a) when to create one?

  - I feel this might be optional, there might be some upper bound
of features that are really "too big" or "too controversial", so
these should undergo the BIP process in all cases, otherwise the
decision might be part of the proposal, the question is how to
define those

 b) what are lifecycle stages of a BIP? How to do transitions
between these stages?

  - From the top of my head this might be:

    a) proposal -- not yet accepted

    b) accepted -- most probably after vote?

    c) in progress -- having assigned JIRA and being worked on

    d) done -- after merge to master

    e) released -- obvious

WDYT?

 Jan

On 1/15/20 8:19 PM, Kenneth Knowles wrote:

Focusing this thread on the BIP process seems wise, without
changing much else in the same thread. I don't think the BIP
process has to do with exactly how design docs are written or
archived, but the ability to *at a glance* understand:

 - what are the high level proposals
 - status of the proposals
 - who to contact
 - how to get to more info (links to design docs, thread, Jiras, etc)

A page with a table on cwiki is common and seems good for this.
How we manage such a table would be a possible next step. I think
they should focus on large changes that need heavyweight process,
so should encourage lightweight creation. I think adding heavy
process to smaller changes would be bad.



I have looked multiple times at other projects (linked in prior
thread and in this thread too but gathering them here)

Spark: https://spark.apache.org/improvement-proposals.html
 - Jira is not good for "at a glance" reading. The title should
have a short and easy to understand paragraph.

Kafka:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
 - Quite a lot of content; I would prefer 10s of proposals. But
it is readable enough. Table lacks important content like links
and summaries.
 - Blends the table with a bunch of header material that IMO ets
in the way

Flink:

https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
 - Looks very similar to Kafka
 - Target Release is too specific, and actual status is missing

Airflow:

https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals
 - seems best organized, and the table has more info
 - having sections for the different status proposals in
different tables is great
 - "InRelease" column is left blank

It seems there is a lot of redundancy with Jira fields - owner,
release, etc. I think that redundancy is good. If it is too much
effort to redundantly manage to write it in the table then it
probably is not appropriate for heavyweight process. Anything
that is one simple task that fits in a Jira that can be passed
around from person to person shouldn't be a BIP. Probably
anything where we can guess the target version isn't big enough
for a BIP.

Kenn

On Thu, Jan 9, 2020 at 7:59 AM Jan Lukavský mailto:je...@seznam.cz>> wrote:

I think that, besides ownership of a feature, a BIP (or
whatever document or process) should contain the following:

 * description of a problem that the improvement addresses  -
this is currently often part of design doc

 * description of multiple possible solutions (if multiple
exist, which is probably mostly the case)

 * justifying choice of a particular solution

 * result of a vote - the vote should cover both (a) do we
don't this feature in the first place and (b) do we accept
   

Re: [DISCUSS] BIP reloaded

[Kenneth Knowles]

These stages sound like a great starting point to me. Would you be the
volunteer to set up a cwiki page for BIPs?

Kenn

On Mon, Jan 20, 2020 at 3:30 AM Jan Lukavský  wrote:

> I agree that we can take inspiration from other projects. Besides the
> organizational part (what should be part of BIP, where to store it, how to
> edit it and how to make it available to the whole community) - for which
> the cwiki might be the best option - there are still open questions about
> the lifecycle of a BIP:
>
>  a) when to create one?
>
>   - I feel this might be optional, there might be some upper bound of
> features that are really "too big" or "too controversial", so these should
> undergo the BIP process in all cases, otherwise the decision might be part
> of the proposal, the question is how to define those
>
>  b) what are lifecycle stages of a BIP? How to do transitions between
> these stages?
>
>   - From the top of my head this might be:
>
> a) proposal -- not yet accepted
>
> b) accepted -- most probably after vote?
>
> c) in progress -- having assigned JIRA and being worked on
>
> d) done -- after merge to master
>
> e) released -- obvious
>
> WDYT?
>
>  Jan
> On 1/15/20 8:19 PM, Kenneth Knowles wrote:
>
> Focusing this thread on the BIP process seems wise, without changing much
> else in the same thread. I don't think the BIP process has to do with
> exactly how design docs are written or archived, but the ability to *at a
> glance* understand:
>
>  - what are the high level proposals
>  - status of the proposals
>  - who to contact
>  - how to get to more info (links to design docs, thread, Jiras, etc)
>
> A page with a table on cwiki is common and seems good for this. How we
> manage such a table would be a possible next step. I think they should
> focus on large changes that need heavyweight process, so should encourage
> lightweight creation. I think adding heavy process to smaller changes would
> be bad.
>
> 
>
> I have looked multiple times at other projects (linked in prior thread and
> in this thread too but gathering them here)
>
> Spark: https://spark.apache.org/improvement-proposals.html
>  - Jira is not good for "at a glance" reading. The title should have a
> short and easy to understand paragraph.
>
> Kafka:
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>  - Quite a lot of content; I would prefer 10s of proposals. But it is
> readable enough. Table lacks important content like links and summaries.
>  - Blends the table with a bunch of header material that IMO ets in the way
>
> Flink:
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>  - Looks very similar to Kafka
>  - Target Release is too specific, and actual status is missing
>
> Airflow:
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals
>  - seems best organized, and the table has more info
>  - having sections for the different status proposals in different tables
> is great
>  - "InRelease" column is left blank
>
> It seems there is a lot of redundancy with Jira fields - owner, release,
> etc. I think that redundancy is good. If it is too much effort to
> redundantly manage to write it in the table then it probably is not
> appropriate for heavyweight process. Anything that is one simple task that
> fits in a Jira that can be passed around from person to person shouldn't be
> a BIP. Probably anything where we can guess the target version isn't big
> enough for a BIP.
>
> Kenn
>
> On Thu, Jan 9, 2020 at 7:59 AM Jan Lukavský  wrote:
>
>> I think that, besides ownership of a feature, a BIP (or whatever document
>> or process) should contain the following:
>>
>>  * description of a problem that the improvement addresses  - this is
>> currently often part of design doc
>>
>>  * description of multiple possible solutions (if multiple exist, which
>> is probably mostly the case)
>>
>>  * justifying choice of a particular solution
>>
>>  * result of a vote - the vote should cover both (a) do we don't this
>> feature in the first place and (b) do we accept the proposed solution
>>
>> This would probably be iterative process involving multiple people,
>> mailing list communication, etc. Pretty much what we do now, just there
>> would be a place to keep track of decisions made throughout the process. I
>> pretty much think that voting on complicated solutions is vital, the soft
>> consensus approach is good for "simple" features (what that means might be
>> subjective), but might fail for features where multiple more or less
>> complex solutions exist. After successful PMC vote, the problem simplifies
>> to reviewing code, the reviewer doesn't have to think about "do we want
>> this feature?". That is given in advance. After we agree on the process and
>> the form it should have I can volunteer to test it by letting proposal of
>> ordered stateful processing pass through it.
>> On 1/9/20 9:11 AM, Alex Van Boxel wrote:
>>
>> 

Re: [DISCUSS] BIP reloaded

[Jan Lukavský]

I agree that we can take inspiration from other projects. Besides the 
organizational part (what should be part of BIP, where to store it, how 
to edit it and how to make it available to the whole community) - for 
which the cwiki might be the best option - there are still open 
questions about the lifecycle of a BIP:


 a) when to create one?

  - I feel this might be optional, there might be some upper bound of 
features that are really "too big" or "too controversial", so these 
should undergo the BIP process in all cases, otherwise the decision 
might be part of the proposal, the question is how to define those


 b) what are lifecycle stages of a BIP? How to do transitions between 
these stages?


  - From the top of my head this might be:

    a) proposal -- not yet accepted

    b) accepted -- most probably after vote?

    c) in progress -- having assigned JIRA and being worked on

    d) done -- after merge to master

    e) released -- obvious

WDYT?

 Jan

On 1/15/20 8:19 PM, Kenneth Knowles wrote:
Focusing this thread on the BIP process seems wise, without changing 
much else in the same thread. I don't think the BIP process has to do 
with exactly how design docs are written or archived, but the ability 
to *at a glance* understand:


 - what are the high level proposals
 - status of the proposals
 - who to contact
 - how to get to more info (links to design docs, thread, Jiras, etc)

A page with a table on cwiki is common and seems good for this. How we 
manage such a table would be a possible next step. I think they should 
focus on large changes that need heavyweight process, so should 
encourage lightweight creation. I think adding heavy process to 
smaller changes would be bad.




I have looked multiple times at other projects (linked in prior thread 
and in this thread too but gathering them here)


Spark: https://spark.apache.org/improvement-proposals.html
 - Jira is not good for "at a glance" reading. The title should have a 
short and easy to understand paragraph.


Kafka: 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
 - Quite a lot of content; I would prefer 10s of proposals. But it is 
readable enough. Table lacks important content like links and summaries.
 - Blends the table with a bunch of header material that IMO ets in 
the way


Flink: 
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

 - Looks very similar to Kafka
 - Target Release is too specific, and actual status is missing

Airflow: 
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals

 - seems best organized, and the table has more info
 - having sections for the different status proposals in different 
tables is great

 - "InRelease" column is left blank

It seems there is a lot of redundancy with Jira fields - owner, 
release, etc. I think that redundancy is good. If it is too much 
effort to redundantly manage to write it in the table then it probably 
is not appropriate for heavyweight process. Anything that is one 
simple task that fits in a Jira that can be passed around from person 
to person shouldn't be a BIP. Probably anything where we can guess the 
target version isn't big enough for a BIP.


Kenn

On Thu, Jan 9, 2020 at 7:59 AM Jan Lukavský > wrote:


I think that, besides ownership of a feature, a BIP (or whatever
document or process) should contain the following:

 * description of a problem that the improvement addresses  - this
is currently often part of design doc

 * description of multiple possible solutions (if multiple exist,
which is probably mostly the case)

 * justifying choice of a particular solution

 * result of a vote - the vote should cover both (a) do we don't
this feature in the first place and (b) do we accept the proposed
solution

This would probably be iterative process involving multiple
people, mailing list communication, etc. Pretty much what we do
now, just there would be a place to keep track of decisions made
throughout the process. I pretty much think that voting on
complicated solutions is vital, the soft consensus approach is
good for "simple" features (what that means might be subjective),
but might fail for features where multiple more or less complex
solutions exist. After successful PMC vote, the problem simplifies
to reviewing code, the reviewer doesn't have to think about "do we
want this feature?". That is given in advance. After we agree on
the process and the form it should have I can volunteer to test it
by letting proposal of ordered stateful processing pass through it.

On 1/9/20 9:11 AM, Alex Van Boxel wrote:

Maybe tweaking the current process a bit is enough. I like the
Docs for having discussions but there no good as a /proper design
document/, for the following reasons:

I see design documents full of discussions and wonder:

 

Re: [DISCUSS] BIP reloaded

[Kenneth Knowles]

Focusing this thread on the BIP process seems wise, without changing much
else in the same thread. I don't think the BIP process has to do with
exactly how design docs are written or archived, but the ability to *at a
glance* understand:

 - what are the high level proposals
 - status of the proposals
 - who to contact
 - how to get to more info (links to design docs, thread, Jiras, etc)

A page with a table on cwiki is common and seems good for this. How we
manage such a table would be a possible next step. I think they should
focus on large changes that need heavyweight process, so should encourage
lightweight creation. I think adding heavy process to smaller changes would
be bad.



I have looked multiple times at other projects (linked in prior thread and
in this thread too but gathering them here)

Spark: https://spark.apache.org/improvement-proposals.html
 - Jira is not good for "at a glance" reading. The title should have a
short and easy to understand paragraph.

Kafka:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
 - Quite a lot of content; I would prefer 10s of proposals. But it is
readable enough. Table lacks important content like links and summaries.
 - Blends the table with a bunch of header material that IMO ets in the way

Flink:
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
 - Looks very similar to Kafka
 - Target Release is too specific, and actual status is missing

Airflow:
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals
 - seems best organized, and the table has more info
 - having sections for the different status proposals in different tables
is great
 - "InRelease" column is left blank

It seems there is a lot of redundancy with Jira fields - owner, release,
etc. I think that redundancy is good. If it is too much effort to
redundantly manage to write it in the table then it probably is not
appropriate for heavyweight process. Anything that is one simple task that
fits in a Jira that can be passed around from person to person shouldn't be
a BIP. Probably anything where we can guess the target version isn't big
enough for a BIP.

Kenn

On Thu, Jan 9, 2020 at 7:59 AM Jan Lukavský  wrote:

> I think that, besides ownership of a feature, a BIP (or whatever document
> or process) should contain the following:
>
>  * description of a problem that the improvement addresses  - this is
> currently often part of design doc
>
>  * description of multiple possible solutions (if multiple exist, which is
> probably mostly the case)
>
>  * justifying choice of a particular solution
>
>  * result of a vote - the vote should cover both (a) do we don't this
> feature in the first place and (b) do we accept the proposed solution
>
> This would probably be iterative process involving multiple people,
> mailing list communication, etc. Pretty much what we do now, just there
> would be a place to keep track of decisions made throughout the process. I
> pretty much think that voting on complicated solutions is vital, the soft
> consensus approach is good for "simple" features (what that means might be
> subjective), but might fail for features where multiple more or less
> complex solutions exist. After successful PMC vote, the problem simplifies
> to reviewing code, the reviewer doesn't have to think about "do we want
> this feature?". That is given in advance. After we agree on the process and
> the form it should have I can volunteer to test it by letting proposal of
> ordered stateful processing pass through it.
> On 1/9/20 9:11 AM, Alex Van Boxel wrote:
>
> Maybe tweaking the current process a bit is enough. I like the Docs for
> having discussions but there no good as a *proper design document*, for
> the following reasons:
>
> I see design documents full of discussions and wonder:
>
>- Who will be the *main owner* and the *co-owners* (meaning people
>that are invested of bringing this forward and can *act* as *reviewers*).
>I think a proposal needs especially this: ownership
>- Lack of visibility of final state? Or is it superseded by another
>proposal. A final state could include the votes...
>- Does the proposal need amendments. An example,  while implementing
>the proposal, we see that something in the design was lacking and needs to
>be added.
>
> So the Docs are great, but maybe we should a few mandatory blocks and a
> few rules:
>
>- *Resolve all discussions* before switching to final state.
>- If new discussions pop up, maybe an amendment needs to be made (or
>correct). Corrections could be added to a *changelog* in the beginning.
>- If a new proposal supersedes on, both should be linked
>- Most importantly: Who can act as *owner* end reviewers for this
>proposal.
>
>
>
>  _/
> _/ Alex Van Boxel
>
>
> On Thu, Jan 9, 2020 at 7:59 AM Kenneth Knowles  wrote:
>
>> It does seem that the community would find this useful. I agree with
>> 

Re: [DISCUSS] BIP reloaded

[Jan Lukavský]

I think that, besides ownership of a feature, a BIP (or whatever 
document or process) should contain the following:


 * description of a problem that the improvement addresses  - this is 
currently often part of design doc


 * description of multiple possible solutions (if multiple exist, which 
is probably mostly the case)


 * justifying choice of a particular solution

 * result of a vote - the vote should cover both (a) do we don't this 
feature in the first place and (b) do we accept the proposed solution


This would probably be iterative process involving multiple people, 
mailing list communication, etc. Pretty much what we do now, just there 
would be a place to keep track of decisions made throughout the process. 
I pretty much think that voting on complicated solutions is vital, the 
soft consensus approach is good for "simple" features (what that means 
might be subjective), but might fail for features where multiple more or 
less complex solutions exist. After successful PMC vote, the problem 
simplifies to reviewing code, the reviewer doesn't have to think about 
"do we want this feature?". That is given in advance. After we agree on 
the process and the form it should have I can volunteer to test it by 
letting proposal of ordered stateful processing pass through it.


On 1/9/20 9:11 AM, Alex Van Boxel wrote:
Maybe tweaking the current process a bit is enough. I like the Docs 
for having discussions but there no good as a /proper design 
document/, for the following reasons:


I see design documents full of discussions and wonder:

  * Who will be the *main owner* and the *co-owners* (meaning people
that are invested of bringing this forward and can *act* as
/reviewers/). I think a proposal needs especially this: ownership
  * Lack of visibility of final state? Or is it superseded by another
proposal. A final state could include the votes...
  * Does the proposal need amendments. An example,  while implementing
the proposal, we see that something in the design was lacking and
needs to be added.

So the Docs are great, but maybe we should a few mandatory blocks and 
a few rules:


  * *Resolve all discussions* before switching to final state.
  * If new discussions pop up, maybe an amendment needs to be made (or
correct). Corrections could be added to a *changelog* in the
beginning.
  * If a new proposal supersedes on, both should be linked
  * Most importantly: Who can act as *owner* end reviewers for this
proposal.



 _/
_/ Alex Van Boxel


On Thu, Jan 9, 2020 at 7:59 AM Kenneth Knowles > wrote:


It does seem that the community would find this useful. I agree
with Robert that it has downsides and it is not appropriate all
the time.

We added https://beam.apache.org/roadmap/ a little while ago. I
think that the granularity of a BIP is about the same as the
granularity of what we would want to show to users on a roadmap on
our public site. So we sort of already have this. Perhaps we want
to formalize changes to the roadmap and only include voted upon
approved BIPs on the roadmap on the web site. The current roadmap
should be viewed as a crowd sourced bootstrap, for sure.

Imagine a roadmap that a company shares with a customer. The most
important thing is to be extremely clear about what is intended to
be built, when it is expected, and how they can follow the
developments. And for the open source community, it should be
clear what they can expect to work on and know that the project /
PMC has agreed on the feature and will not push back after some
effort has been put into it.

Kenn

On Tue, Dec 17, 2019 at 11:07 AM Jan Lukavský mailto:je...@seznam.cz>> wrote:

Hi,

I feel a "soft consensus" :) that people see some benefits of
introducing (possibly optional) process of proposing new features.

I think that in order to proceed with this we need to agree on
goals that we want to achieve. Whether the process should or
should not be optional, which form it should have, and answers
on all these other questions could be answered after that.

So, I'll try to state some issues I see with our current
approach, please feel free to correct any of them, or add any
other:

 - due to the "soft consensus" approach, we actually delegate
the final responsibility of "feature acceptance" to
reviewer(s) - these might or might not be happy with that

 - by splitting this into
first-consensus-then-implementation-then-review approach, we
remove the burden of responsibility of respective feature from
reviewers - they can focus only on the main purpose of the
review - that is verifying the quality of code

 - as mentioned before, this brings better visibility to
(core) features

 - and last but not least makes it 

Re: [DISCUSS] BIP reloaded

[Alex Van Boxel]

Maybe tweaking the current process a bit is enough. I like the Docs for
having discussions but there no good as a *proper design document*, for the
following reasons:

I see design documents full of discussions and wonder:

   - Who will be the *main owner* and the *co-owners* (meaning people that
   are invested of bringing this forward and can *act* as *reviewers*). I
   think a proposal needs especially this: ownership
   - Lack of visibility of final state? Or is it superseded by another
   proposal. A final state could include the votes...
   - Does the proposal need amendments. An example,  while implementing the
   proposal, we see that something in the design was lacking and needs to be
   added.

So the Docs are great, but maybe we should a few mandatory blocks and a few
rules:

   - *Resolve all discussions* before switching to final state.
   - If new discussions pop up, maybe an amendment needs to be made (or
   correct). Corrections could be added to a *changelog* in the beginning.
   - If a new proposal supersedes on, both should be linked
   - Most importantly: Who can act as *owner* end reviewers for this
   proposal.



 _/
_/ Alex Van Boxel


On Thu, Jan 9, 2020 at 7:59 AM Kenneth Knowles  wrote:

> It does seem that the community would find this useful. I agree with
> Robert that it has downsides and it is not appropriate all the time.
>
> We added https://beam.apache.org/roadmap/ a little while ago. I think
> that the granularity of a BIP is about the same as the granularity of what
> we would want to show to users on a roadmap on our public site. So we sort
> of already have this. Perhaps we want to formalize changes to the roadmap
> and only include voted upon approved BIPs on the roadmap on the web site.
> The current roadmap should be viewed as a crowd sourced bootstrap, for sure.
>
> Imagine a roadmap that a company shares with a customer. The most
> important thing is to be extremely clear about what is intended to be
> built, when it is expected, and how they can follow the developments. And
> for the open source community, it should be clear what they can expect to
> work on and know that the project / PMC has agreed on the feature and will
> not push back after some effort has been put into it.
>
> Kenn
>
> On Tue, Dec 17, 2019 at 11:07 AM Jan Lukavský  wrote:
>
>> Hi,
>>
>> I feel a "soft consensus" :) that people see some benefits of introducing
>> (possibly optional) process of proposing new features.
>>
>> I think that in order to proceed with this we need to agree on goals that
>> we want to achieve. Whether the process should or should not be optional,
>> which form it should have, and answers on all these other questions could
>> be answered after that.
>>
>> So, I'll try to state some issues I see with our current approach, please
>> feel free to correct any of them, or add any other:
>>
>>  - due to the "soft consensus" approach, we actually delegate the final
>> responsibility of "feature acceptance" to reviewer(s) - these might or
>> might not be happy with that
>>
>>  - by splitting this into first-consensus-then-implementation-then-review
>> approach, we remove the burden of responsibility of respective feature from
>> reviewers - they can focus only on the main purpose of the review - that is
>> verifying the quality of code
>>
>>  - as mentioned before, this brings better visibility to (core) features
>>
>>  - and last but not least makes it possible to prioritize work and build
>> more complex long-term goals
>>
>> I think it is essential to have a consensus on whether or not these are
>> some points we want to target (that is, we see our current approach as
>> sub-optimal in these areas) or not.
>>
>> Jan
>> On 12/17/19 7:08 PM, Pablo Estrada wrote:
>>
>> It seems that lots of people see benefit in a more formalized BIP
>> process. I think that makes sense, though I'd like to give people the
>> freedom to choose the medium for their design discussions.
>>
>> The projects I'm aware of usually do this through wiki-type mediums. We
>> have cwiki, though lots of people like working with Gdocs' collaboration
>> features. Are there other mediums that could be used for this?
>>
>> A possible implementation is: We could keep cwiki as the 'index' - so
>> anyone proposing a new BIP would have to add a new BIP entry in the cwiki,
>> but they'd be free to link to a Gdoc from there, or to develop the proposal
>> in the cwiki entry itself.
>>
>> Thoughts?
>> Best
>> -P.
>>
>> On Tue, Dec 17, 2019 at 9:14 AM Maximilian Michels 
>> wrote:
>>
>>> The main benefit of BIPs I see is the visibility they create for the
>>> project users and contributors.
>>>
>>> Right now, we have a long unordnered list of design documents. Some of
>>> the documents are not even in that list. With BIPs, we would end up with
>>> an ordered list "BIP-1, BIP-2, .." which reflects important design
>>> decisions over time.
>>>
>>> Simply assigning an id, makes it a lot more formal. In my eyes, the id

Re: [DISCUSS] BIP reloaded

[Kenneth Knowles]

It does seem that the community would find this useful. I agree with Robert
that it has downsides and it is not appropriate all the time.

We added https://beam.apache.org/roadmap/ a little while ago. I think that
the granularity of a BIP is about the same as the granularity of what we
would want to show to users on a roadmap on our public site. So we sort of
already have this. Perhaps we want to formalize changes to the roadmap and
only include voted upon approved BIPs on the roadmap on the web site. The
current roadmap should be viewed as a crowd sourced bootstrap, for sure.

Imagine a roadmap that a company shares with a customer. The most important
thing is to be extremely clear about what is intended to be built, when it
is expected, and how they can follow the developments. And for the open
source community, it should be clear what they can expect to work on and
know that the project / PMC has agreed on the feature and will not push
back after some effort has been put into it.

Kenn

On Tue, Dec 17, 2019 at 11:07 AM Jan Lukavský  wrote:

> Hi,
>
> I feel a "soft consensus" :) that people see some benefits of introducing
> (possibly optional) process of proposing new features.
>
> I think that in order to proceed with this we need to agree on goals that
> we want to achieve. Whether the process should or should not be optional,
> which form it should have, and answers on all these other questions could
> be answered after that.
>
> So, I'll try to state some issues I see with our current approach, please
> feel free to correct any of them, or add any other:
>
>  - due to the "soft consensus" approach, we actually delegate the final
> responsibility of "feature acceptance" to reviewer(s) - these might or
> might not be happy with that
>
>  - by splitting this into first-consensus-then-implementation-then-review
> approach, we remove the burden of responsibility of respective feature from
> reviewers - they can focus only on the main purpose of the review - that is
> verifying the quality of code
>
>  - as mentioned before, this brings better visibility to (core) features
>
>  - and last but not least makes it possible to prioritize work and build
> more complex long-term goals
>
> I think it is essential to have a consensus on whether or not these are
> some points we want to target (that is, we see our current approach as
> sub-optimal in these areas) or not.
>
> Jan
> On 12/17/19 7:08 PM, Pablo Estrada wrote:
>
> It seems that lots of people see benefit in a more formalized BIP process.
> I think that makes sense, though I'd like to give people the freedom to
> choose the medium for their design discussions.
>
> The projects I'm aware of usually do this through wiki-type mediums. We
> have cwiki, though lots of people like working with Gdocs' collaboration
> features. Are there other mediums that could be used for this?
>
> A possible implementation is: We could keep cwiki as the 'index' - so
> anyone proposing a new BIP would have to add a new BIP entry in the cwiki,
> but they'd be free to link to a Gdoc from there, or to develop the proposal
> in the cwiki entry itself.
>
> Thoughts?
> Best
> -P.
>
> On Tue, Dec 17, 2019 at 9:14 AM Maximilian Michels  wrote:
>
>> The main benefit of BIPs I see is the visibility they create for the
>> project users and contributors.
>>
>> Right now, we have a long unordnered list of design documents. Some of
>> the documents are not even in that list. With BIPs, we would end up with
>> an ordered list "BIP-1, BIP-2, .." which reflects important design
>> decisions over time.
>>
>> Simply assigning an id, makes it a lot more formal. In my eyes, the id
>> assignment would also require that you communicate the changes in a way
>> that the community can accept the proposal, preferably via lazy
>> consensus. All in all, this could help communicate changes in Beam better.
>>
>> JIRA, on the other hand, contains concrete implementation steps and all
>> kinds of other changes.
>>
>> Cheers,
>> Max
>>
>> On 16.12.19 21:41, Robert Bradshaw wrote:
>> > Additional process is a two-edged sword: it can help move stuff
>> > forward, to the correct decision, but it can also add significant
>> > overhead.
>> >
>> > I think there are many proposals for which the existing processes of
>> > deriving consensus (over email, possibly followed by a formal vote or
>> > lazy consensus) are sufficient. However, sometimes they're not.
>> > Specifically, for long-term roadmaps, it would be useful to have them
>> > in a standard place that can be tracked and understood (I don't think
>> > we've been able to use JIRA effectively for this here). I also think
>> > there are some proposals that reach a certain level of complexity that
>> > trying to address them by occasionally responding to email threads as
>> > they come up is insufficient. For these latter, I think there is a
>> > need for commitment for a group of people in the community to commit
>> > to clearly defining and driving a 

Re: [DISCUSS] BIP reloaded

[Jan Lukavský]

Hi,

I feel a "soft consensus" :) that people see some benefits of 
introducing (possibly optional) process of proposing new features.


I think that in order to proceed with this we need to agree on goals 
that we want to achieve. Whether the process should or should not be 
optional, which form it should have, and answers on all these other 
questions could be answered after that.


So, I'll try to state some issues I see with our current approach, 
please feel free to correct any of them, or add any other:


 - due to the "soft consensus" approach, we actually delegate the final 
responsibility of "feature acceptance" to reviewer(s) - these might or 
might not be happy with that


 - by splitting this into 
first-consensus-then-implementation-then-review approach, we remove the 
burden of responsibility of respective feature from reviewers - they can 
focus only on the main purpose of the review - that is verifying the 
quality of code


 - as mentioned before, this brings better visibility to (core) features

 - and last but not least makes it possible to prioritize work and 
build more complex long-term goals


I think it is essential to have a consensus on whether or not these are 
some points we want to target (that is, we see our current approach as 
sub-optimal in these areas) or not.


Jan

On 12/17/19 7:08 PM, Pablo Estrada wrote:
It seems that lots of people see benefit in a more formalized BIP 
process. I think that makes sense, though I'd like to give people the 
freedom to choose the medium for their design discussions.


The projects I'm aware of usually do this through wiki-type mediums. 
We have cwiki, though lots of people like working with Gdocs' 
collaboration features. Are there other mediums that could be used for 
this?


A possible implementation is: We could keep cwiki as the 'index' - so 
anyone proposing a new BIP would have to add a new BIP entry in the 
cwiki, but they'd be free to link to a Gdoc from there, or to develop 
the proposal in the cwiki entry itself.


Thoughts?
Best
-P.

On Tue, Dec 17, 2019 at 9:14 AM Maximilian Michels > wrote:


The main benefit of BIPs I see is the visibility they create for the
project users and contributors.

Right now, we have a long unordnered list of design documents.
Some of
the documents are not even in that list. With BIPs, we would end
up with
an ordered list "BIP-1, BIP-2, .." which reflects important design
decisions over time.

Simply assigning an id, makes it a lot more formal. In my eyes,
the id
assignment would also require that you communicate the changes in
a way
that the community can accept the proposal, preferably via lazy
consensus. All in all, this could help communicate changes in Beam
better.

JIRA, on the other hand, contains concrete implementation steps
and all
kinds of other changes.

Cheers,
Max

On 16.12.19 21:41, Robert Bradshaw wrote:
> Additional process is a two-edged sword: it can help move stuff
> forward, to the correct decision, but it can also add significant
> overhead.
>
> I think there are many proposals for which the existing processes of
> deriving consensus (over email, possibly followed by a formal
vote or
> lazy consensus) are sufficient. However, sometimes they're not.
> Specifically, for long-term roadmaps, it would be useful to have
them
> in a standard place that can be tracked and understood (I don't
think
> we've been able to use JIRA effectively for this here). I also think
> there are some proposals that reach a certain level of
complexity that
> trying to address them by occasionally responding to email
threads as
> they come up is insufficient. For these latter, I think there is a
> need for commitment for a group of people in the community to commit
> to clearly defining and driving a solution to the problem via a more
> formal process. Often the one making the proposal has sufficient
> motivation, but sometimes what lacks is be (non-sporadic) investment
> by those trying to understand, evaluate, and incorporate the
proposal.
>
> So I'm (strongly) +1 for exploring a more formal process, but -1 on
> requiring it.
>
> On Sun, Dec 15, 2019 at 1:07 AM Jan Lukavský mailto:je...@seznam.cz>> wrote:
>>
>> Hi,
>>
>> thanks for reactions so far. I agree that there are many
questions that have to be clarified. I'd propose to split this
into two parts:
>>
>>   a) first reach a consensus that we want this process in the
first place
>>
>>   b) after that, we need to clarify all the details - that will
probably be somewhat iterative procedure
>>
>> I'm not sure if there is something more we need to clarify
before we can cast a vote on (a).
>>
>> Thoughts?
>>
>>   Jan
>>
>> On 12/10/19 3:46 PM, 

Re: [DISCUSS] BIP reloaded

[Pablo Estrada]

It seems that lots of people see benefit in a more formalized BIP process.
I think that makes sense, though I'd like to give people the freedom to
choose the medium for their design discussions.

The projects I'm aware of usually do this through wiki-type mediums. We
have cwiki, though lots of people like working with Gdocs' collaboration
features. Are there other mediums that could be used for this?

A possible implementation is: We could keep cwiki as the 'index' - so
anyone proposing a new BIP would have to add a new BIP entry in the cwiki,
but they'd be free to link to a Gdoc from there, or to develop the proposal
in the cwiki entry itself.

Thoughts?
Best
-P.

On Tue, Dec 17, 2019 at 9:14 AM Maximilian Michels  wrote:

> The main benefit of BIPs I see is the visibility they create for the
> project users and contributors.
>
> Right now, we have a long unordnered list of design documents. Some of
> the documents are not even in that list. With BIPs, we would end up with
> an ordered list "BIP-1, BIP-2, .." which reflects important design
> decisions over time.
>
> Simply assigning an id, makes it a lot more formal. In my eyes, the id
> assignment would also require that you communicate the changes in a way
> that the community can accept the proposal, preferably via lazy
> consensus. All in all, this could help communicate changes in Beam better.
>
> JIRA, on the other hand, contains concrete implementation steps and all
> kinds of other changes.
>
> Cheers,
> Max
>
> On 16.12.19 21:41, Robert Bradshaw wrote:
> > Additional process is a two-edged sword: it can help move stuff
> > forward, to the correct decision, but it can also add significant
> > overhead.
> >
> > I think there are many proposals for which the existing processes of
> > deriving consensus (over email, possibly followed by a formal vote or
> > lazy consensus) are sufficient. However, sometimes they're not.
> > Specifically, for long-term roadmaps, it would be useful to have them
> > in a standard place that can be tracked and understood (I don't think
> > we've been able to use JIRA effectively for this here). I also think
> > there are some proposals that reach a certain level of complexity that
> > trying to address them by occasionally responding to email threads as
> > they come up is insufficient. For these latter, I think there is a
> > need for commitment for a group of people in the community to commit
> > to clearly defining and driving a solution to the problem via a more
> > formal process. Often the one making the proposal has sufficient
> > motivation, but sometimes what lacks is be (non-sporadic) investment
> > by those trying to understand, evaluate, and incorporate the proposal.
> >
> > So I'm (strongly) +1 for exploring a more formal process, but -1 on
> > requiring it.
> >
> > On Sun, Dec 15, 2019 at 1:07 AM Jan Lukavský  wrote:
> >>
> >> Hi,
> >>
> >> thanks for reactions so far. I agree that there are many questions that
> have to be clarified. I'd propose to split this into two parts:
> >>
> >>   a) first reach a consensus that we want this process in the first
> place
> >>
> >>   b) after that, we need to clarify all the details - that will
> probably be somewhat iterative procedure
> >>
> >> I'm not sure if there is something more we need to clarify before we
> can cast a vote on (a).
> >>
> >> Thoughts?
> >>
> >>   Jan
> >>
> >> On 12/10/19 3:46 PM, Łukasz Gajowy wrote:
> >>
> >> +1 for formalizing the process, enhancing it and documenting clearly.
> >>
> >> I noticed that Apache Airflow has a cool way of both creating AIPs and
> keeping track of all of them. There is a "Create new AIP" button on their
> Confluence. This way, no AIP gets lost and all are kept in one place.
> Please keep in mind that this is also the problem we want to solve in Beam
> and try to keep track of all the documents we have so far*. It's certainly
> good to solve that problem too, if possible.
> >>
> >> Also the AIP structure is something that I find nice - There's place
> for all additional resources, JIRAs, discussion in comments and state of
> the proposal. Even if we don't choose to use Confluence, we definitely
> could use a similar template with all that information for our google docs
> proposals or any other tool we stick to.
> >>
> >> Thanks!
> >>
> >> *thank you, Ismael and Alexey, for all the reminders under the
> proposals to add them to Confluence list! :)
> >>
> >> wt., 10 gru 2019 o 13:29 jincheng sun 
> napisał(a):
> >>>
> >>> Thanks for bring up this discussion Jan!
> >>>
> >>> +1 for cearly define BIP for beam.
> >>>
> >>> And I think would be nice to initialize a concept document for BIP.
> Just a reminder: the document may contains:
> >>>
> >>> - How many kinds of improvement in beam.
> >>> - What kind of improvement should to create a BIP.
> >>> - What should be included in a BIP.
> >>> - Who can create the BIP.
> >>> - Who can participate in the discussion of BIP and who can vote for
> BIP.
> >>> - What are the 

Re: [DISCUSS] BIP reloaded

[Maximilian Michels]

The main benefit of BIPs I see is the visibility they create for the 
project users and contributors.


Right now, we have a long unordnered list of design documents. Some of 
the documents are not even in that list. With BIPs, we would end up with 
an ordered list "BIP-1, BIP-2, .." which reflects important design 
decisions over time.


Simply assigning an id, makes it a lot more formal. In my eyes, the id 
assignment would also require that you communicate the changes in a way 
that the community can accept the proposal, preferably via lazy 
consensus. All in all, this could help communicate changes in Beam better.


JIRA, on the other hand, contains concrete implementation steps and all 
kinds of other changes.


Cheers,
Max

On 16.12.19 21:41, Robert Bradshaw wrote:

Additional process is a two-edged sword: it can help move stuff
forward, to the correct decision, but it can also add significant
overhead.

I think there are many proposals for which the existing processes of
deriving consensus (over email, possibly followed by a formal vote or
lazy consensus) are sufficient. However, sometimes they're not.
Specifically, for long-term roadmaps, it would be useful to have them
in a standard place that can be tracked and understood (I don't think
we've been able to use JIRA effectively for this here). I also think
there are some proposals that reach a certain level of complexity that
trying to address them by occasionally responding to email threads as
they come up is insufficient. For these latter, I think there is a
need for commitment for a group of people in the community to commit
to clearly defining and driving a solution to the problem via a more
formal process. Often the one making the proposal has sufficient
motivation, but sometimes what lacks is be (non-sporadic) investment
by those trying to understand, evaluate, and incorporate the proposal.

So I'm (strongly) +1 for exploring a more formal process, but -1 on
requiring it.

On Sun, Dec 15, 2019 at 1:07 AM Jan Lukavský  wrote:


Hi,

thanks for reactions so far. I agree that there are many questions that have to 
be clarified. I'd propose to split this into two parts:

  a) first reach a consensus that we want this process in the first place

  b) after that, we need to clarify all the details - that will probably be 
somewhat iterative procedure

I'm not sure if there is something more we need to clarify before we can cast a 
vote on (a).

Thoughts?

  Jan

On 12/10/19 3:46 PM, Łukasz Gajowy wrote:

+1 for formalizing the process, enhancing it and documenting clearly.

I noticed that Apache Airflow has a cool way of both creating AIPs and keeping track of 
all of them. There is a "Create new AIP" button on their Confluence. This way, 
no AIP gets lost and all are kept in one place. Please keep in mind that this is also the 
problem we want to solve in Beam and try to keep track of all the documents we have so 
far*. It's certainly good to solve that problem too, if possible.

Also the AIP structure is something that I find nice - There's place for all 
additional resources, JIRAs, discussion in comments and state of the proposal. 
Even if we don't choose to use Confluence, we definitely could use a similar 
template with all that information for our google docs proposals or any other 
tool we stick to.

Thanks!

*thank you, Ismael and Alexey, for all the reminders under the proposals to add 
them to Confluence list! :)

wt., 10 gru 2019 o 13:29 jincheng sun  napisał(a):


Thanks for bring up this discussion Jan!

+1 for cearly define BIP for beam.

And I think would be nice to initialize a concept document for BIP. Just a 
reminder: the document may contains:

- How many kinds of improvement in beam.
- What kind of improvement should to create a BIP.
- What should be included in a BIP.
- Who can create the BIP.
- Who can participate in the discussion of BIP and who can vote for BIP.
- What are the possible limitations of BiP, such as whether it is necessary to 
complete the dev of BIP  in one release.
- How to track a BIP.

Here is a question: I found out a policy[1] in beam, but only contains the 
poilcy of release , my question is does beam have something called Bylaws? 
Similar as Flink[1].

Anyway, I like your proposals Jan :)

Best,
Jincheng
[1] https://beam.apache.org/community/policies/
[2] 
https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals


David Morávek  于2019年12月10日周二 下午2:33写道：


Hi Jan,

I think this is more pretty much what we currently do, just a little bit more 
transparent for the community. If the process is standardized, it can open 
doors for bigger contributions from people not familiar with the process. Also 
it's way easier to track progress of BIPs, than documents linked from the 
mailing list.

Big +1 ;)

D.

On Sun, Dec 8, 2019 at 12:42 PM Jan Lukavský  wrote:


Hi,

I'd like to revive a discussion that was taken some year and a half ago
[1], which included a concept of "BIP" (Beam 

Re: [DISCUSS] BIP reloaded

[Robert Bradshaw]

Additional process is a two-edged sword: it can help move stuff
forward, to the correct decision, but it can also add significant
overhead.

I think there are many proposals for which the existing processes of
deriving consensus (over email, possibly followed by a formal vote or
lazy consensus) are sufficient. However, sometimes they're not.
Specifically, for long-term roadmaps, it would be useful to have them
in a standard place that can be tracked and understood (I don't think
we've been able to use JIRA effectively for this here). I also think
there are some proposals that reach a certain level of complexity that
trying to address them by occasionally responding to email threads as
they come up is insufficient. For these latter, I think there is a
need for commitment for a group of people in the community to commit
to clearly defining and driving a solution to the problem via a more
formal process. Often the one making the proposal has sufficient
motivation, but sometimes what lacks is be (non-sporadic) investment
by those trying to understand, evaluate, and incorporate the proposal.

So I'm (strongly) +1 for exploring a more formal process, but -1 on
requiring it.

On Sun, Dec 15, 2019 at 1:07 AM Jan Lukavský  wrote:
>
> Hi,
>
> thanks for reactions so far. I agree that there are many questions that have 
> to be clarified. I'd propose to split this into two parts:
>
>  a) first reach a consensus that we want this process in the first place
>
>  b) after that, we need to clarify all the details - that will probably be 
> somewhat iterative procedure
>
> I'm not sure if there is something more we need to clarify before we can cast 
> a vote on (a).
>
> Thoughts?
>
>  Jan
>
> On 12/10/19 3:46 PM, Łukasz Gajowy wrote:
>
> +1 for formalizing the process, enhancing it and documenting clearly.
>
> I noticed that Apache Airflow has a cool way of both creating AIPs and 
> keeping track of all of them. There is a "Create new AIP" button on their 
> Confluence. This way, no AIP gets lost and all are kept in one place. Please 
> keep in mind that this is also the problem we want to solve in Beam and try 
> to keep track of all the documents we have so far*. It's certainly good to 
> solve that problem too, if possible.
>
> Also the AIP structure is something that I find nice - There's place for all 
> additional resources, JIRAs, discussion in comments and state of the 
> proposal. Even if we don't choose to use Confluence, we definitely could use 
> a similar template with all that information for our google docs proposals or 
> any other tool we stick to.
>
> Thanks!
>
> *thank you, Ismael and Alexey, for all the reminders under the proposals to 
> add them to Confluence list! :)
>
> wt., 10 gru 2019 o 13:29 jincheng sun  napisał(a):
>>
>> Thanks for bring up this discussion Jan!
>>
>> +1 for cearly define BIP for beam.
>>
>> And I think would be nice to initialize a concept document for BIP. Just a 
>> reminder: the document may contains:
>>
>> - How many kinds of improvement in beam.
>> - What kind of improvement should to create a BIP.
>> - What should be included in a BIP.
>> - Who can create the BIP.
>> - Who can participate in the discussion of BIP and who can vote for BIP.
>> - What are the possible limitations of BiP, such as whether it is necessary 
>> to complete the dev of BIP  in one release.
>> - How to track a BIP.
>>
>> Here is a question: I found out a policy[1] in beam, but only contains the 
>> poilcy of release , my question is does beam have something called Bylaws? 
>> Similar as Flink[1].
>>
>> Anyway, I like your proposals Jan :)
>>
>> Best,
>> Jincheng
>> [1] https://beam.apache.org/community/policies/
>> [2] 
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals
>>
>>
>> David Morávek  于2019年12月10日周二 下午2:33写道：
>>>
>>> Hi Jan,
>>>
>>> I think this is more pretty much what we currently do, just a little bit 
>>> more transparent for the community. If the process is standardized, it can 
>>> open doors for bigger contributions from people not familiar with the 
>>> process. Also it's way easier to track progress of BIPs, than documents 
>>> linked from the mailing list.
>>>
>>> Big +1 ;)
>>>
>>> D.
>>>
>>> On Sun, Dec 8, 2019 at 12:42 PM Jan Lukavský  wrote:

 Hi,

 I'd like to revive a discussion that was taken some year and a half ago
 [1], which included a concept of "BIP" (Beam Improvement Proposal) - an
 equivalent of "FLIP" (flink), "KIP" (kafka), "SPIP" (spark), and so on.

 The discussion then ended without any (public) conclusion, so I'd like
 to pick up from there. There were questions related to:

   a) how does the concept of BIP differ from simple plain JIRA?

   b) what does it bring to the community?

 I'd like to outline my point of view on both of these aspects (they are
 related).

 BIP differs from JIRA by definition of a process:

 BIP -> vote -> 

Re: [DISCUSS] BIP reloaded

[Jan Lukavský]

Hi,

thanks for reactions so far. I agree that there are many questions that 
have to be clarified. I'd propose to split this into two parts:


 a) first reach a consensus that we want this process in the first place

 b) after that, we need to clarify all the details - that will probably 
be somewhat iterative procedure


I'm not sure if there is something more we need to clarify before we can 
cast a vote on (a).


Thoughts?

 Jan

On 12/10/19 3:46 PM, Łukasz Gajowy wrote:

+1 for formalizing the process, enhancing it and documenting clearly.

I noticed that Apache Airflow has a cool way of both creating AIPs and 
keeping track of all of them. There is a "Create new AIP" 
 
button on their Confluence. This way, no AIP gets lost and all are 
kept in one place. Please keep in mind that this is also the problem 
we want to solve in Beam and try to keep track of all the documents we 
have so far*. It's certainly good to solve that problem too, if possible.


Also the AIP structure 
 
is something that I find nice - There's place for all additional 
resources, JIRAs, discussion in comments and state of the proposal. 
Even if we don't choose to use Confluence, we definitely could use a 
similar template with all that information for our google docs 
proposals or any other tool we stick to.


Thanks!

*thank you, Ismael and Alexey, for all the reminders under the 
proposals to add them to Confluence list 
! :)


wt., 10 gru 2019 o 13:29 jincheng sun > napisał(a):


Thanks for bring up this discussion Jan!

+1 for cearly define BIP for beam.

And I think would be nice to initialize a concept document for
BIP. Just a reminder: the document may contains:

- How many kinds of improvement in beam.
- What kind of improvement should to create a BIP.
- What should be included in a BIP.
- Who can create the BIP.
- Who can participate in the discussion of BIP and who can vote
for BIP.
- What are the possible limitations of BiP, such as whether it is
necessary to complete the dev of BIP  in one release.
- How to track a BIP.

Here is a question: I found out a policy[1] in beam, but only
contains the poilcy of release , my question is does beam have
something called Bylaws? Similar as Flink[1].

Anyway, I like your proposals Jan :)

Best,
Jincheng
[1] https://beam.apache.org/community/policies/
[2]

https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals


David Morávek mailto:david.mora...@gmail.com>> 于2019年12月10日周二 下午2:33写道：

Hi Jan,

I think this is more pretty much what we currently do, just a
little bit more transparent for the community. If the process
is standardized, it can open doors for bigger contributions
from people not familiar with the process. Also it's way
easier to track progress of BIPs, than documents linked from
the mailing list.

Big +1 ;)

D.

On Sun, Dec 8, 2019 at 12:42 PM Jan Lukavský mailto:je...@seznam.cz>> wrote:

Hi,

I'd like to revive a discussion that was taken some year
and a half ago
[1], which included a concept of "BIP" (Beam Improvement
Proposal) - an
equivalent of "FLIP" (flink), "KIP" (kafka), "SPIP"
(spark), and so on.

The discussion then ended without any (public) conclusion,
so I'd like
to pick up from there. There were questions related to:

  a) how does the concept of BIP differ from simple plain
JIRA?

  b) what does it bring to the community?

I'd like to outline my point of view on both of these
aspects (they are
related).

BIP differs from JIRA by definition of a process:

    BIP -> vote -> consensus -> JIRA -> implementation

This process (although it might seem a little unnecessary
formal) brings
the following benefits:

  i) improves community's overall awareness of planned and
in-progress
features

  ii) makes it possible to prioritize long-term goals
(create "roadmap"
that was mentioned in the referred thread)

  iii) by casting explicit vote on each improvement
proposal diminishes
the probability of wasted work - as opposed to our current
state, where
it is hard to tell when there is a consensus and what
actions need to be
done in order to reach one if there isn't

  iv) 

Re: [DISCUSS] BIP reloaded

[Łukasz Gajowy]

+1 for formalizing the process, enhancing it and documenting clearly.

I noticed that Apache Airflow has a cool way of both creating AIPs and
keeping track of all of them. There is a "Create new AIP"

button on their Confluence. This way, no AIP gets lost and all are kept in
one place. Please keep in mind that this is also the problem we want to
solve in Beam and try to keep track of all the documents we have so far*.
It's certainly good to solve that problem too, if possible.

Also the AIP structure

is something that I find nice - There's place for all additional resources,
JIRAs, discussion in comments and state of the proposal. Even if we don't
choose to use Confluence, we definitely could use a similar template with
all that information for our google docs proposals or any other tool we
stick to.

Thanks!

*thank you, Ismael and Alexey, for all the reminders under the proposals to
add them to Confluence list
! :)

wt., 10 gru 2019 o 13:29 jincheng sun  napisał(a):

> Thanks for bring up this discussion Jan!
>
> +1 for cearly define BIP for beam.
>
> And I think would be nice to initialize a concept document for BIP. Just a
> reminder: the document may contains:
>
> - How many kinds of improvement in beam.
> - What kind of improvement should to create a BIP.
> - What should be included in a BIP.
> - Who can create the BIP.
> - Who can participate in the discussion of BIP and who can vote for BIP.
> - What are the possible limitations of BiP, such as whether it is
> necessary to complete the dev of BIP  in one release.
> - How to track a BIP.
>
> Here is a question: I found out a policy[1] in beam, but only contains the
> poilcy of release , my question is does beam have something called Bylaws?
> Similar as Flink[1].
>
> Anyway, I like your proposals Jan :)
>
> Best,
> Jincheng
> [1] https://beam.apache.org/community/policies/
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals
>
>
> David Morávek  于2019年12月10日周二 下午2:33写道：
>
>> Hi Jan,
>>
>> I think this is more pretty much what we currently do, just a little bit
>> more transparent for the community. If the process is standardized, it can
>> open doors for bigger contributions from people not familiar with the
>> process. Also it's way easier to track progress of BIPs, than documents
>> linked from the mailing list.
>>
>> Big +1 ;)
>>
>> D.
>>
>> On Sun, Dec 8, 2019 at 12:42 PM Jan Lukavský  wrote:
>>
>>> Hi,
>>>
>>> I'd like to revive a discussion that was taken some year and a half ago
>>> [1], which included a concept of "BIP" (Beam Improvement Proposal) - an
>>> equivalent of "FLIP" (flink), "KIP" (kafka), "SPIP" (spark), and so on.
>>>
>>> The discussion then ended without any (public) conclusion, so I'd like
>>> to pick up from there. There were questions related to:
>>>
>>>   a) how does the concept of BIP differ from simple plain JIRA?
>>>
>>>   b) what does it bring to the community?
>>>
>>> I'd like to outline my point of view on both of these aspects (they are
>>> related).
>>>
>>> BIP differs from JIRA by definition of a process:
>>>
>>> BIP -> vote -> consensus -> JIRA -> implementation
>>>
>>> This process (although it might seem a little unnecessary formal) brings
>>> the following benefits:
>>>
>>>   i) improves community's overall awareness of planned and in-progress
>>> features
>>>
>>>   ii) makes it possible to prioritize long-term goals (create "roadmap"
>>> that was mentioned in the referred thread)
>>>
>>>   iii) by casting explicit vote on each improvement proposal diminishes
>>> the probability of wasted work - as opposed to our current state, where
>>> it is hard to tell when there is a consensus and what actions need to be
>>> done in order to reach one if there isn't
>>>
>>>   iv) BIPs that eventually pass a vote can be regarded as "to be
>>> included in some short term" and so new BIPs can build upon them,
>>> without the risk of having to be redefined if their dependency for
>>> whatever reason don't make it to the implementation
>>>
>>> Although this "process" might look rigid and corporate, it actually
>>> brings better transparency and overall community health. This is
>>> especially important as the community grows and becomes more and more
>>> distributed. There are many, many open questions in this proposal that
>>> need to be clarified, my current intent is to grab a grasp about how the
>>> community feels about this.
>>>
>>> Looking forward to any comments,
>>>
>>>   Jan
>>>
>>> [1]
>>>
>>> https://lists.apache.org/thread.html/4e1fffa2fde8e750c6d769bf4335853ad05b360b8bd248ad119cc185%40%3Cdev.beam.apache.org%3E
>>>
>>>

Re: [DISCUSS] BIP reloaded

[jincheng sun]

Thanks for bring up this discussion Jan!

+1 for cearly define BIP for beam.

And I think would be nice to initialize a concept document for BIP. Just a
reminder: the document may contains:

- How many kinds of improvement in beam.
- What kind of improvement should to create a BIP.
- What should be included in a BIP.
- Who can create the BIP.
- Who can participate in the discussion of BIP and who can vote for BIP.
- What are the possible limitations of BiP, such as whether it is necessary
to complete the dev of BIP  in one release.
- How to track a BIP.

Here is a question: I found out a policy[1] in beam, but only contains the
poilcy of release , my question is does beam have something called Bylaws?
Similar as Flink[1].

Anyway, I like your proposals Jan :)

Best,
Jincheng
[1] https://beam.apache.org/community/policies/
[2]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals


David Morávek  于2019年12月10日周二 下午2:33写道：

> Hi Jan,
>
> I think this is more pretty much what we currently do, just a little bit
> more transparent for the community. If the process is standardized, it can
> open doors for bigger contributions from people not familiar with the
> process. Also it's way easier to track progress of BIPs, than documents
> linked from the mailing list.
>
> Big +1 ;)
>
> D.
>
> On Sun, Dec 8, 2019 at 12:42 PM Jan Lukavský  wrote:
>
>> Hi,
>>
>> I'd like to revive a discussion that was taken some year and a half ago
>> [1], which included a concept of "BIP" (Beam Improvement Proposal) - an
>> equivalent of "FLIP" (flink), "KIP" (kafka), "SPIP" (spark), and so on.
>>
>> The discussion then ended without any (public) conclusion, so I'd like
>> to pick up from there. There were questions related to:
>>
>>   a) how does the concept of BIP differ from simple plain JIRA?
>>
>>   b) what does it bring to the community?
>>
>> I'd like to outline my point of view on both of these aspects (they are
>> related).
>>
>> BIP differs from JIRA by definition of a process:
>>
>> BIP -> vote -> consensus -> JIRA -> implementation
>>
>> This process (although it might seem a little unnecessary formal) brings
>> the following benefits:
>>
>>   i) improves community's overall awareness of planned and in-progress
>> features
>>
>>   ii) makes it possible to prioritize long-term goals (create "roadmap"
>> that was mentioned in the referred thread)
>>
>>   iii) by casting explicit vote on each improvement proposal diminishes
>> the probability of wasted work - as opposed to our current state, where
>> it is hard to tell when there is a consensus and what actions need to be
>> done in order to reach one if there isn't
>>
>>   iv) BIPs that eventually pass a vote can be regarded as "to be
>> included in some short term" and so new BIPs can build upon them,
>> without the risk of having to be redefined if their dependency for
>> whatever reason don't make it to the implementation
>>
>> Although this "process" might look rigid and corporate, it actually
>> brings better transparency and overall community health. This is
>> especially important as the community grows and becomes more and more
>> distributed. There are many, many open questions in this proposal that
>> need to be clarified, my current intent is to grab a grasp about how the
>> community feels about this.
>>
>> Looking forward to any comments,
>>
>>   Jan
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/4e1fffa2fde8e750c6d769bf4335853ad05b360b8bd248ad119cc185%40%3Cdev.beam.apache.org%3E
>>
>>

Re: [DISCUSS] BIP reloaded

[David Morávek]

Hi Jan,

I think this is more pretty much what we currently do, just a little bit
more transparent for the community. If the process is standardized, it can
open doors for bigger contributions from people not familiar with the
process. Also it's way easier to track progress of BIPs, than documents
linked from the mailing list.

Big +1 ;)

D.

On Sun, Dec 8, 2019 at 12:42 PM Jan Lukavský  wrote:

> Hi,
>
> I'd like to revive a discussion that was taken some year and a half ago
> [1], which included a concept of "BIP" (Beam Improvement Proposal) - an
> equivalent of "FLIP" (flink), "KIP" (kafka), "SPIP" (spark), and so on.
>
> The discussion then ended without any (public) conclusion, so I'd like
> to pick up from there. There were questions related to:
>
>   a) how does the concept of BIP differ from simple plain JIRA?
>
>   b) what does it bring to the community?
>
> I'd like to outline my point of view on both of these aspects (they are
> related).
>
> BIP differs from JIRA by definition of a process:
>
> BIP -> vote -> consensus -> JIRA -> implementation
>
> This process (although it might seem a little unnecessary formal) brings
> the following benefits:
>
>   i) improves community's overall awareness of planned and in-progress
> features
>
>   ii) makes it possible to prioritize long-term goals (create "roadmap"
> that was mentioned in the referred thread)
>
>   iii) by casting explicit vote on each improvement proposal diminishes
> the probability of wasted work - as opposed to our current state, where
> it is hard to tell when there is a consensus and what actions need to be
> done in order to reach one if there isn't
>
>   iv) BIPs that eventually pass a vote can be regarded as "to be
> included in some short term" and so new BIPs can build upon them,
> without the risk of having to be redefined if their dependency for
> whatever reason don't make it to the implementation
>
> Although this "process" might look rigid and corporate, it actually
> brings better transparency and overall community health. This is
> especially important as the community grows and becomes more and more
> distributed. There are many, many open questions in this proposal that
> need to be clarified, my current intent is to grab a grasp about how the
> community feels about this.
>
> Looking forward to any comments,
>
>   Jan
>
> [1]
>
> https://lists.apache.org/thread.html/4e1fffa2fde8e750c6d769bf4335853ad05b360b8bd248ad119cc185%40%3Cdev.beam.apache.org%3E
>
>