Re: [DISCUSS] BIP reloaded

[Kenneth Knowles]

Sounds great. If you scrape recent dev@ for proposals that are not yet
implemented, I think you will find some, and you could ask them to add as a
BIP if they are still interested.

Kenn

On Sat, Feb 1, 2020 at 1:11 AM Jan Lukavský  wrote:

> Hi Kenn,
>
> yes, I can do that. I think that there should be at least one first BIP, I
> can try to setup one. But (as opposed to my previous proposal) I'll try to
> setup a fresh one, not the one of [BEAM-8550], because that one already has
> a PR and rebasing the PR on master for such a long period (and it is
> likely, that final polishing of the BIP process will take several more
> months) starts to be costly. I have in mind two fresh candidates, so I'll
> pick one of them. I think that only setuping a cwiki would not start the
> process, we need a real-life example of a BIP included in that.
>
> Does that sound ok?
>
>  Jan
> On 2/1/20 5:55 AM, Kenneth Knowles wrote:
>
> These stages sound like a great starting point to me. Would you be the
> volunteer to set up a cwiki page for BIPs?
>
> Kenn
>
> On Mon, Jan 20, 2020 at 3:30 AM Jan Lukavský  wrote:
>
>> I agree that we can take inspiration from other projects. Besides the
>> organizational part (what should be part of BIP, where to store it, how to
>> edit it and how to make it available to the whole community) - for which
>> the cwiki might be the best option - there are still open questions about
>> the lifecycle of a BIP:
>>
>>  a) when to create one?
>>
>>   - I feel this might be optional, there might be some upper bound of
>> features that are really "too big" or "too controversial", so these should
>> undergo the BIP process in all cases, otherwise the decision might be part
>> of the proposal, the question is how to define those
>>
>>  b) what are lifecycle stages of a BIP? How to do transitions between
>> these stages?
>>
>>   - From the top of my head this might be:
>>
>> a) proposal -- not yet accepted
>>
>> b) accepted -- most probably after vote?
>>
>> c) in progress -- having assigned JIRA and being worked on
>>
>> d) done -- after merge to master
>>
>> e) released -- obvious
>>
>> WDYT?
>>
>>  Jan
>> On 1/15/20 8:19 PM, Kenneth Knowles wrote:
>>
>> Focusing this thread on the BIP process seems wise, without changing much
>> else in the same thread. I don't think the BIP process has to do with
>> exactly how design docs are written or archived, but the ability to *at a
>> glance* understand:
>>
>>  - what are the high level proposals
>>  - status of the proposals
>>  - who to contact
>>  - how to get to more info (links to design docs, thread, Jiras, etc)
>>
>> A page with a table on cwiki is common and seems good for this. How we
>> manage such a table would be a possible next step. I think they should
>> focus on large changes that need heavyweight process, so should encourage
>> lightweight creation. I think adding heavy process to smaller changes would
>> be bad.
>>
>> 
>>
>> I have looked multiple times at other projects (linked in prior thread
>> and in this thread too but gathering them here)
>>
>> Spark: https://spark.apache.org/improvement-proposals.html
>>  - Jira is not good for "at a glance" reading. The title should have a
>> short and easy to understand paragraph.
>>
>> Kafka:
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>>  - Quite a lot of content; I would prefer 10s of proposals. But it is
>> readable enough. Table lacks important content like links and summaries.
>>  - Blends the table with a bunch of header material that IMO ets in the
>> way
>>
>> Flink:
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>>  - Looks very similar to Kafka
>>  - Target Release is too specific, and actual status is missing
>>
>> Airflow:
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals
>>  - seems best organized, and the table has more info
>>  - having sections for the different status proposals in different tables
>> is great
>>  - "InRelease" column is left blank
>>
>> It seems there is a lot of redundancy with Jira fields - owner, release,
>> etc. I think that redundancy is good. If it is too much effort to
>> redundantly manage to write it in the table then it probably is not
>> appropriate for heavyweight process. Anything that is one simple task that
>> fits in a Jira that can be passed around from person to person shouldn't be
>> a BIP. Probably anything where we can guess the target version isn't big
>> enough for a BIP.
>>
>> Kenn
>>
>> On Thu, Jan 9, 2020 at 7:59 AM Jan Lukavský  wrote:
>>
>>> I think that, besides ownership of a feature, a BIP (or whatever
>>> document or process) should contain the following:
>>>
>>>  * description of a problem that the improvement addresses  - this is
>>> currently often part of design doc
>>>
>>>  * description of multiple possible solutions (if multiple exist, which
>>> is probably mostly the case)
>>>
>>>  * justifying 

Re: [DISCUSSION] Improve release notes by adding a change list file

[Chad Dombrova]

In case it's of any use, there's a tool called towncrier[1] to help compile
changelog fragments and compile them at time of delivery.

I came across this when working on the python-attrs[2] project, which has
some good documentation for contributors on how to use it:
https://www.attrs.org/en/stable/contributing.html#changelog



[1] https://github.com/hawkowl/towncrier
[2] https://github.com/python-attrs/attrs


On Fri, Jan 31, 2020 at 5:09 PM Ahmet Altay  wrote:

> Thank you for the quick responses. I sent out
> https://github.com/apache/beam/pull/10743 to make this change. Please
> provide feedback or directly edit the PR.
>
> On Fri, Jan 31, 2020 at 3:58 PM Robert Bradshaw 
> wrote:
>
>> Yes, yes, yes! This is the one model of release notes that I've
>> actually seen work well at scale.
>>
>>
>> https://lists.apache.org/thread.html/41e03ace17dbcccf7e267ba6d538736b2a99a8e73e7fb45702766b17%40%3Cdev.beam.apache.org%3E
>>
>> Let's make it happen.
>>
>> On Fri, Jan 31, 2020 at 3:47 PM Robert Burke  wrote:
>> >
>> > I like this suggestion, Jira titles and commit summaries don't
>> necessarily reflect the user impact for a given change (or set of changes).
>> Being able to see the Forest instead of the trees.
>> >
>> > On Fri, Jan 31, 2020, 3:37 PM Kenneth Knowles  wrote:
>> >>
>> >> +1
>> >>
>> >> This is a great idea. Hope it can lead to higher-value view of
>> relevant changes.
>> >>
>> >> I like it being in the root of the repo, so it lives next to the code.
>> >>
>> >> Since the website is also markdown, it could be copied over directly
>> at release time, so it can be browsed there, too.
>> >>
>> >> Kenn
>> >>
>> >> On Fri, Jan 31, 2020 at 3:16 PM Ahmet Altay  wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> We currently have two major ways to communicate changes in a release:
>> >>> - A blog post, to highlight major changes in the release. (Example
>> for 2.17: [1])
>> >>> - JIRA release notes pages listing all issues tagged for a specific
>> release. (Example for 2.17 [2]).
>> >>>
>> >>> There are a few issues with this process:
>> >>> - It is difficult for the release manager to know what is important,
>> what is a breaking change, what is dependency change etc. For example,
>> there were more than 150 Jira issues tagged for 2.17 release.
>> >>> - Release blog has many items, and does not necessarily communicate
>> important changes. It is difficult for users to discover major changes
>> short of going through a large list.
>> >>> - People involved in authoring or reviewing a PRs usually have the
>> most context about the change, and they are not necessarily involved in the
>> release process to provide this additional information.
>> >>>
>> >>> Would it be helpful if we maintain a simple change list file and
>> update it as part of the PRs with noteworthy changes? Release managers
>> could use this information as is in their blog posts (or link to it). Users
>> will have a single place to find highlights from various versions.
>> >>>
>> >>> Concretely, I am proposing:
>> >>> - Adding a CHANGES file to the root of the repository. (Name could be
>> anything, TFX uses RELEASE.md in their repo. [3])
>> >>> - Ask PR authors to update this file as part of their PR whenever it
>> makes sense
>> >>> - Reference this file during the release process, and a new section
>> for the next release after each release.
>> >>>
>> >>> Ahmet
>> >>>
>> >>> [1] https://beam.apache.org/blog/2020/01/06/beam-2.17.0.html
>> >>> [2]
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12345970=12319527
>> >>> [3] https://github.com/tensorflow/tfx/blob/master/RELEASE.md
>>
>

Re: [DISCUSS] BIP reloaded

[Jan Lukavský]

Hi Kenn,

yes, I can do that. I think that there should be at least one first BIP, 
I can try to setup one. But (as opposed to my previous proposal) I'll 
try to setup a fresh one, not the one of [BEAM-8550], because that one 
already has a PR and rebasing the PR on master for such a long period 
(and it is likely, that final polishing of the BIP process will take 
several more months) starts to be costly. I have in mind two fresh 
candidates, so I'll pick one of them. I think that only setuping a cwiki 
would not start the process, we need a real-life example of a BIP 
included in that.


Does that sound ok?

 Jan

On 2/1/20 5:55 AM, Kenneth Knowles wrote:
These stages sound like a great starting point to me. Would you be the 
volunteer to set up a cwiki page for BIPs?


Kenn

On Mon, Jan 20, 2020 at 3:30 AM Jan Lukavský > wrote:


I agree that we can take inspiration from other projects. Besides
the organizational part (what should be part of BIP, where to
store it, how to edit it and how to make it available to the whole
community) - for which the cwiki might be the best option - there
are still open questions about the lifecycle of a BIP:

 a) when to create one?

  - I feel this might be optional, there might be some upper bound
of features that are really "too big" or "too controversial", so
these should undergo the BIP process in all cases, otherwise the
decision might be part of the proposal, the question is how to
define those

 b) what are lifecycle stages of a BIP? How to do transitions
between these stages?

  - From the top of my head this might be:

    a) proposal -- not yet accepted

    b) accepted -- most probably after vote?

    c) in progress -- having assigned JIRA and being worked on

    d) done -- after merge to master

    e) released -- obvious

WDYT?

 Jan

On 1/15/20 8:19 PM, Kenneth Knowles wrote:

Focusing this thread on the BIP process seems wise, without
changing much else in the same thread. I don't think the BIP
process has to do with exactly how design docs are written or
archived, but the ability to *at a glance* understand:

 - what are the high level proposals
 - status of the proposals
 - who to contact
 - how to get to more info (links to design docs, thread, Jiras, etc)

A page with a table on cwiki is common and seems good for this.
How we manage such a table would be a possible next step. I think
they should focus on large changes that need heavyweight process,
so should encourage lightweight creation. I think adding heavy
process to smaller changes would be bad.



I have looked multiple times at other projects (linked in prior
thread and in this thread too but gathering them here)

Spark: https://spark.apache.org/improvement-proposals.html
 - Jira is not good for "at a glance" reading. The title should
have a short and easy to understand paragraph.

Kafka:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
 - Quite a lot of content; I would prefer 10s of proposals. But
it is readable enough. Table lacks important content like links
and summaries.
 - Blends the table with a bunch of header material that IMO ets
in the way

Flink:

https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
 - Looks very similar to Kafka
 - Target Release is too specific, and actual status is missing

Airflow:

https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals
 - seems best organized, and the table has more info
 - having sections for the different status proposals in
different tables is great
 - "InRelease" column is left blank

It seems there is a lot of redundancy with Jira fields - owner,
release, etc. I think that redundancy is good. If it is too much
effort to redundantly manage to write it in the table then it
probably is not appropriate for heavyweight process. Anything
that is one simple task that fits in a Jira that can be passed
around from person to person shouldn't be a BIP. Probably
anything where we can guess the target version isn't big enough
for a BIP.

Kenn

On Thu, Jan 9, 2020 at 7:59 AM Jan Lukavský mailto:je...@seznam.cz>> wrote:

I think that, besides ownership of a feature, a BIP (or
whatever document or process) should contain the following:

 * description of a problem that the improvement addresses  -
this is currently often part of design doc

 * description of multiple possible solutions (if multiple
exist, which is probably mostly the case)

 * justifying choice of a particular solution

 * result of a vote - the vote should cover both (a) do we
don't this feature in the first place and (b) do we accept