Shall we use "tenacity" library to help deflake some of Python tests using retry logic?

[Valentyn Tymofieiev]

I have been looking at a few test flakes in Python SDK recently, and some
of them can benefit from a simple retry logic. See PR #7455  for an
example[1].

I would not recommend retrying by default for all tests, or mechanically
adding a retry to every test that we see flaking: some legitimate bugs may
manifest in flakes that happen rarely, and a retry logic will hide them
from us.

However, in some tests we can consider adding retries, retries with a
back-off or retries only on particular exceptions. tenacity[2] offers
several decorators for this purpose. It is available under Apache 2.0
license on PyPi [3], and is being maintained.

What does the community think about adding this library as a test-only
dependency to Python SDK?

Thanks,
Valentyn

[1] https://github.com/apache/beam/pull/7455
[2] https://github.com/jd/tenacity
[3] https://pypi.org/project/tenacity/

Re: Add all tests to release validation

[Kenneth Knowles]

What do you think about crowd-sourcing?

1. Fix Version = 2.10.0
2. If assigned, ping ticket and maybe assignee, unassign if unresponsive
3. If unassigned, assign it to yourself while thinking about it
4. If you can route it a bit closer to someone who might know, great
5. If it doesn't look like a blocker (after routing best you can), Fix
Version = 2.11.0

I think this has enough mutexes that there should be no duplicated work if
it is followed. And every step is a standard use of Fix Version and
Assignee so there's not really special policy needed.

Kenn

On Thu, Jan 10, 2019 at 4:25 PM Mikhail Gryzykhin  wrote:

> +1
>
> Although we should be cautious when enabling this policy. We have decent
> backlog of bugs that we need to plumb through.
>
> --Mikhail
>
> Have feedback ?
>
>
> On Thu, Jan 10, 2019 at 11:44 AM Scott Wegner  wrote:
>
>> +1, this sounds good to me.
>>
>> I believe the next step would be to open a PR to add this to the release
>> guide:
>> https://github.com/apache/beam/blob/master/website/src/contribute/release-guide.md
>>
>> On Wed, Jan 9, 2019 at 12:04 PM Sam Rohde  wrote:
>>
>>> Cool, thanks for all of the replies. Does this summary sound reasonable?
>>>
>>> *Problem:* there are a number of failing tests (including flaky) that
>>> don't get looked at, and aren't necessarily green upon cutting a new Beam
>>> release.
>>>
>>> *Proposed Solution:*
>>>
>>>- Add all tests to the release validation
>>>- For all failing tests (including flaky) create a JIRA attached to
>>>the Beam release and add to the "test-failures" component*
>>>- If a test is continuously failing
>>>  - fix it
>>>  - add fix to release
>>>  - close out JIRA
>>>   - If a test is flaky
>>>  - try and fix it
>>>  - If fixed
>>> - add fix to release
>>> - close out JIRA
>>>  - else
>>> - manually test it
>>> - modify "Fix Version" to next release
>>>  - The release validation can continue when all JIRAs are
>>>closed out.
>>>
>>> *Why this is an improvement:*
>>>
>>>- Ensures that every test is a valid signal (as opposed to disabling
>>>failing tests)
>>>- Creates an incentive to automate tests (no longer on the hook to
>>>manually test)
>>>- Creates a forcing-function to fix flaky tests (once fixed, no
>>>longer needs to be manually tested)
>>>- Ensures that every failing test gets looked at
>>>
>>> *Why this may not be an improvement:*
>>>
>>>- More effort for release validation
>>>- May slow down release velocity
>>>
>>> * for brevity, this might be better to create a JIRA per component
>>> containing a summary of failing tests
>>>
>>>
>>> -Sam
>>>
>>>
>>>
>>>
>>> On Tue, Jan 8, 2019 at 10:25 AM Ahmet Altay  wrote:
>>>


 On Tue, Jan 8, 2019 at 8:25 AM Kenneth Knowles  wrote:

>
>
> On Tue, Jan 8, 2019 at 7:52 AM Scott Wegner  wrote:
>
>> For reference, there are currently 34 unresolved JIRA issues under
>> the test-failures component [1].
>>
>> [1]
>> https://issues.apache.org/jira/browse/BEAM-6280?jql=project%20%3D%20BEAM%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
>>
>
> And there are 19 labeled with flake or sickbay:
> https://issues.apache.org/jira/issues/?filter=12343195
>
>
>> On Mon, Jan 7, 2019 at 4:03 PM Ahmet Altay  wrote:
>>
>>> This is a a good idea. Some suggestions:
>>> - It would be nicer if we can figure out process to act on flaky
>>> test more frequently than releases.
>>>
>>
> Any ideas? We could just have some cadence and try to establish the
> practice of having a deflake thread every couple of weeks? How about we 
> add
> it to release verification as a first step and then continue to discuss?
>

 Sounds great. I do not know enough JIRA, but I am hoping that a
 solution can come in the form of tooling. If we could configure JIRA with
 SLOs per issue type, we could have customized reports on which issues are
 not getting enough attention and then do a load balance among us.


>
> - Another improvement in the process would be having actual owners of
>>> issues rather than auto assigned component owners. A few folks have 100+
>>> assigned issues. Unassigning those issues, and finding owners who would
>>> have time to work on identified flaky tests would be helpful.
>>>
>>
> Yikes. Two issues here:
>
>  - sounds like Jira component owners aren't really working for us as a
> first point of contact for triage
>  - a person shouldn't really have more than 5 Jira assigned, or if you
> get really loose maybe 20 (I am guilty of having 30 at this moment...)
>
> Maybe this is one or two separate threads?
>

 I can fork

Re: Add all tests to release validation

[Mikhail Gryzykhin]

+1

Although we should be cautious when enabling this policy. We have decent
backlog of bugs that we need to plumb through.

--Mikhail

Have feedback ?


On Thu, Jan 10, 2019 at 11:44 AM Scott Wegner  wrote:

> +1, this sounds good to me.
>
> I believe the next step would be to open a PR to add this to the release
> guide:
> https://github.com/apache/beam/blob/master/website/src/contribute/release-guide.md
>
> On Wed, Jan 9, 2019 at 12:04 PM Sam Rohde  wrote:
>
>> Cool, thanks for all of the replies. Does this summary sound reasonable?
>>
>> *Problem:* there are a number of failing tests (including flaky) that
>> don't get looked at, and aren't necessarily green upon cutting a new Beam
>> release.
>>
>> *Proposed Solution:*
>>
>>- Add all tests to the release validation
>>- For all failing tests (including flaky) create a JIRA attached to
>>the Beam release and add to the "test-failures" component*
>>- If a test is continuously failing
>>  - fix it
>>  - add fix to release
>>  - close out JIRA
>>   - If a test is flaky
>>  - try and fix it
>>  - If fixed
>> - add fix to release
>> - close out JIRA
>>  - else
>> - manually test it
>> - modify "Fix Version" to next release
>>  - The release validation can continue when all JIRAs are closed
>>out.
>>
>> *Why this is an improvement:*
>>
>>- Ensures that every test is a valid signal (as opposed to disabling
>>failing tests)
>>- Creates an incentive to automate tests (no longer on the hook to
>>manually test)
>>- Creates a forcing-function to fix flaky tests (once fixed, no
>>longer needs to be manually tested)
>>- Ensures that every failing test gets looked at
>>
>> *Why this may not be an improvement:*
>>
>>- More effort for release validation
>>- May slow down release velocity
>>
>> * for brevity, this might be better to create a JIRA per component
>> containing a summary of failing tests
>>
>>
>> -Sam
>>
>>
>>
>>
>> On Tue, Jan 8, 2019 at 10:25 AM Ahmet Altay  wrote:
>>
>>>
>>>
>>> On Tue, Jan 8, 2019 at 8:25 AM Kenneth Knowles  wrote:
>>>


 On Tue, Jan 8, 2019 at 7:52 AM Scott Wegner  wrote:

> For reference, there are currently 34 unresolved JIRA issues under the
> test-failures component [1].
>
> [1]
> https://issues.apache.org/jira/browse/BEAM-6280?jql=project%20%3D%20BEAM%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
>

 And there are 19 labeled with flake or sickbay:
 https://issues.apache.org/jira/issues/?filter=12343195


> On Mon, Jan 7, 2019 at 4:03 PM Ahmet Altay  wrote:
>
>> This is a a good idea. Some suggestions:
>> - It would be nicer if we can figure out process to act on flaky test
>> more frequently than releases.
>>
>
 Any ideas? We could just have some cadence and try to establish the
 practice of having a deflake thread every couple of weeks? How about we add
 it to release verification as a first step and then continue to discuss?

>>>
>>> Sounds great. I do not know enough JIRA, but I am hoping that a solution
>>> can come in the form of tooling. If we could configure JIRA with SLOs per
>>> issue type, we could have customized reports on which issues are not
>>> getting enough attention and then do a load balance among us.
>>>
>>>

 - Another improvement in the process would be having actual owners of
>> issues rather than auto assigned component owners. A few folks have 100+
>> assigned issues. Unassigning those issues, and finding owners who would
>> have time to work on identified flaky tests would be helpful.
>>
>
 Yikes. Two issues here:

  - sounds like Jira component owners aren't really working for us as a
 first point of contact for triage
  - a person shouldn't really have more than 5 Jira assigned, or if you
 get really loose maybe 20 (I am guilty of having 30 at this moment...)

 Maybe this is one or two separate threads?

>>>
>>> I can fork this to another thread. I think both issues are related
>>> because components owners are more likely to be in this situaion. I agree
>>> with assessment of two issues.
>>>
>>>

 Kenn


>
>>
>> On Mon, Jan 7, 2019 at 3:45 PM Kenneth Knowles 
>> wrote:
>>
>>> I love this idea. It can easily feel like bugs filed for Jenkins
>>> flakes/failures just get lost if there is no process for looking them 
>>> over
>>> regularly.
>>>
>>> I would suggest that test failures / flakes all get filed with Fix
>>> Version = whatever release is next. Then at release time we can triage 
>>> the
>>> list, making sure none might be a symptom of something that should block
>>> the release. One modifi

Re: Add all tests to release validation

[Scott Wegner]

+1, this sounds good to me.

I believe the next step would be to open a PR to add this to the release
guide:
https://github.com/apache/beam/blob/master/website/src/contribute/release-guide.md

On Wed, Jan 9, 2019 at 12:04 PM Sam Rohde  wrote:

> Cool, thanks for all of the replies. Does this summary sound reasonable?
>
> *Problem:* there are a number of failing tests (including flaky) that
> don't get looked at, and aren't necessarily green upon cutting a new Beam
> release.
>
> *Proposed Solution:*
>
>- Add all tests to the release validation
>- For all failing tests (including flaky) create a JIRA attached to
>the Beam release and add to the "test-failures" component*
>- If a test is continuously failing
>  - fix it
>  - add fix to release
>  - close out JIRA
>   - If a test is flaky
>  - try and fix it
>  - If fixed
> - add fix to release
> - close out JIRA
>  - else
> - manually test it
> - modify "Fix Version" to next release
>  - The release validation can continue when all JIRAs are closed
>out.
>
> *Why this is an improvement:*
>
>- Ensures that every test is a valid signal (as opposed to disabling
>failing tests)
>- Creates an incentive to automate tests (no longer on the hook to
>manually test)
>- Creates a forcing-function to fix flaky tests (once fixed, no longer
>needs to be manually tested)
>- Ensures that every failing test gets looked at
>
> *Why this may not be an improvement:*
>
>- More effort for release validation
>- May slow down release velocity
>
> * for brevity, this might be better to create a JIRA per component
> containing a summary of failing tests
>
>
> -Sam
>
>
>
>
> On Tue, Jan 8, 2019 at 10:25 AM Ahmet Altay  wrote:
>
>>
>>
>> On Tue, Jan 8, 2019 at 8:25 AM Kenneth Knowles  wrote:
>>
>>>
>>>
>>> On Tue, Jan 8, 2019 at 7:52 AM Scott Wegner  wrote:
>>>
 For reference, there are currently 34 unresolved JIRA issues under the
 test-failures component [1].

 [1]
 https://issues.apache.org/jira/browse/BEAM-6280?jql=project%20%3D%20BEAM%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC

>>>
>>> And there are 19 labeled with flake or sickbay:
>>> https://issues.apache.org/jira/issues/?filter=12343195
>>>
>>>
 On Mon, Jan 7, 2019 at 4:03 PM Ahmet Altay  wrote:

> This is a a good idea. Some suggestions:
> - It would be nicer if we can figure out process to act on flaky test
> more frequently than releases.
>

>>> Any ideas? We could just have some cadence and try to establish the
>>> practice of having a deflake thread every couple of weeks? How about we add
>>> it to release verification as a first step and then continue to discuss?
>>>
>>
>> Sounds great. I do not know enough JIRA, but I am hoping that a solution
>> can come in the form of tooling. If we could configure JIRA with SLOs per
>> issue type, we could have customized reports on which issues are not
>> getting enough attention and then do a load balance among us.
>>
>>
>>>
>>> - Another improvement in the process would be having actual owners of
> issues rather than auto assigned component owners. A few folks have 100+
> assigned issues. Unassigning those issues, and finding owners who would
> have time to work on identified flaky tests would be helpful.
>

>>> Yikes. Two issues here:
>>>
>>>  - sounds like Jira component owners aren't really working for us as a
>>> first point of contact for triage
>>>  - a person shouldn't really have more than 5 Jira assigned, or if you
>>> get really loose maybe 20 (I am guilty of having 30 at this moment...)
>>>
>>> Maybe this is one or two separate threads?
>>>
>>
>> I can fork this to another thread. I think both issues are related
>> because components owners are more likely to be in this situaion. I agree
>> with assessment of two issues.
>>
>>
>>>
>>> Kenn
>>>
>>>

>
> On Mon, Jan 7, 2019 at 3:45 PM Kenneth Knowles 
> wrote:
>
>> I love this idea. It can easily feel like bugs filed for Jenkins
>> flakes/failures just get lost if there is no process for looking them 
>> over
>> regularly.
>>
>> I would suggest that test failures / flakes all get filed with Fix
>> Version = whatever release is next. Then at release time we can triage 
>> the
>> list, making sure none might be a symptom of something that should block
>> the release. One modification to your proposal is that after manual
>> verification that it is safe to release I would move Fix Version to the
>> next release instead of closing, unless the issue really is fixed or
>> otherwise not reproducible.
>>
>> For automation, I wonder if there's something automatic already
>> available somewhere that would:
>>
>>  - mark the J

Re: [DISCUSS] (Forked thread) Beam issue triage & assignees

[Scott Wegner]

+1

> 3) Ensure that each component's unresolved issues get looked at regularly

This is ideal, but I also don't know how to get to this state. Starting
with clear component ownership and expectations will help. If the triaging
process is well-defined, then members of the community can help for any
components which need additional support.

On Thu, Jan 10, 2019 at 12:21 AM Mikhail Gryzykhin <
gryzykhin.mikh...@gmail.com> wrote:

> +1 to keep issues unassigned and reevaluate backlog from time to time.
>
> We can also auto-unassign if there was no activity on ticket for N days.
> Or we can have auto-mailed report that highlights stale assigned issues.
>
> On Thu, Jan 10, 2019 at 12:10 AM Robert Bradshaw 
> wrote:
>
>> On Thu, Jan 10, 2019 at 3:20 AM Ahmet Altay  wrote:
>> >
>> > I agree with the proposals here. Initial state of "Needs Review" and
>> blocking releases on untriaged issues will ensure that we will at least
>> look at every new issue once.
>>
>> +1.
>>
>> I'm more ambivalent about closing stale issues. Unlike PRs, issues can
>> be filed as "we should (not forget to) do this" much sooner than
>> they're actively worked on.
>>
>> > On Wed, Jan 9, 2019 at 10:30 AM Maximilian Michels 
>> wrote:
>> >>
>> >> Hi Kenn,
>> >>
>> >> As your data shows, default-assigning issues to a single person does
>> not
>> >> automatically solve triaging issues. Quite the contrary, it hides the
>> triage
>> >> status of an issue.
>> >>
>> >>  From the perspective of the Flink Runner, we used to auto-assign but
>> we got rid
>> >> of this. Instead, we monitor the newly coming issues and take actions.
>> We also
>> >> go through the old ones occasionally. I believe that works fine for us.
>> >>
>> >> The Flink project itself also does not default-assign, newly created
>> issues are
>> >> unassigned. There are component leads overseeing issues. There is no
>> guarantee
>> >> that every issue gets triaged.
>> >>
>> >> "Needs Triage" or or "Needs Review" (seems easier to understand of
>> non-native
>> >> speakers) sounds like a good addition, but it will not solve the
>> problem that
>> >> issues need to be curated and maintained after the initial triage. For
>> example,
>> >> I've seen issues not closed after they have been fixed via a PR.
>> However, "Needs
>> >> Triage" will ensure that all issues get looked at. This could be
>> helpful for
>> >> releases, if not-yet-triaged issues are looked at early enough.
>> >>
>> >> I'd suggest to:
>> >>
>> >> 1) Change new issues to be Unassigned and to be in status "Needs
>> Review"
>> >> 2) Remove Assignees from all not-being-worked-on issues
>> >
>> >
>> > For the existing issues, I suggest unassign all issues assigned to
>> people with > N issues for a large N. Something like 30, > %1 of all
>> issues. There are also issues assigned to people who are no longer active
>> in the community. We could unassign those as well.
>> >
>> > Another issue is average age for open issues is also ever growing and
>> is over > 300 days now. It would be nice if we can have an equivalent of
>> stale bot for PRs for JIRAs, unassigning/closing stale issues periodically.
>> >
>> >>
>> >> 3) Ensure that each component's unresolved issues get looked at
>> regularly
>> >>
>> >> I suppose how to do 3) effectively is an open question. I currently
>> don't see a
>> >> better way as to oversee each component by multiple committers,
>> similar to the
>> >> OWNERS idea that we once had. I think we should move the OWNERS
>> information to a
>> >> Wiki page to make ownership/maintainership more transparent and easier
>> to change.
>> >
>> >
>> > I think this is a good idea. We can also divide components even more
>> and there will be more people looking at smaller components each. We could
>> also consider defining SLO type metrics especially for high priority
>> issues, and present those in a dashboard. That way we can collectively see
>> the overall health of our triaging efforts and prevent important issues
>> from being forgotten.
>> >
>> >>
>> >>
>> >> Thanks,
>> >> Max
>> >>
>> >> On 08.01.19 16:30, Kenneth Knowles wrote:
>> >> > Forking discussion on
>> >> >
>> >> >   - Some folks have 100+ issues assigned
>> >> >   - Jira components might not be the best way to get things triaged
>> >> >
>> >> > I just ran a built-in Jira report to summarize how many tickets are
>> claimed by
>> >> > different users [1]. It does look like component owners tend to
>> accumulate
>> >> > issues and they are not likely to get addressed.
>> >> >
>> >> > So what do we want and how can we make it happen? Here is what I
>> want, personally:
>> >> >
>> >> >   - every new issue gets looked at by someone who can decide the
>> right
>> >> > component, who to ping, etc
>> >> >   - no person is assigned an issue that they are not active on
>> >> >   - we don't release with potential bugs waiting to be triaged
>> >> >
>> >> > The current status it that issues get assigned to a component owner
>> when filed.
>> >> > The component ow

Re: Load testing on DirectRunner

[Andrew Pilloud]

My default advice here is to use the Direct Runner for small smoke tests,
and use the Flink LocalRunner for larger datasets that can still be run
locally. As Reuven points out, the Direct Runner is more of a validation
test itself, it does many things designed to test pipelines for the worst
combinations of conditions they may encounter on other runners.

Andrew

On Thu, Jan 10, 2019 at 10:10 AM Reuven Lax  wrote:

> The Direct Runner as currently implemented is purposely inefficient. It
> was designed for testing, and therefore does many things that are meant to
> expose bugs in user pipelines (e.g. randomly sorting PCollections,
> serializing/deserializing every element, etc.). So it's not surprising that
> it doesn't behave well under load tests.
>
> Reuven
>
> On Thu, Jan 10, 2019 at 5:55 AM Katarzyna Kucharczyk <
> ka.kucharc...@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> My name is Kasia and I contribute to Beam's tests. Currently, I am
>> working with Łukasz Gajowy on load tests and we created Jenkins
>> configuration to run Synthetic Sources test on DirectRunner. It was decided
>> to generate 1 000 000 000 records (bytes) for a small suite (details you
>> can find in this proposal [1] ). Running this test on the Beam’s Jenkins is
>> causing runtime exception with the message:
>> "java.lang.OutOfMemoryError: GC overhead limit exceeded".
>>
>> Of course, this is not a surprise since it's a lot of data. That's why I
>> am asking for your advice/opinion:
>> Do you think if this test should be smaller? On the other hand, if it's
>> going to be smaller would it be still worth testing as a load test?
>> Maybe it would be better to wait for the UniversalLocalRunner instead and
>> use it while it's there? What is the status of ULR?  Do you know if the ULR
>> will replace DirectRunner?
>>
>> I created an issue [2] with details of this problem where you can find
>> the link to the example of a failing job.
>>
>> Thanks,
>> Kasia
>>
>> [1] https://s.apache.org/load-test-basic-operations
>> 
>> [2] https://issues.apache.org/jira/browse/BEAM-6351
>>
>

Re: Load testing on DirectRunner

[Rui Wang]

Agree with Reuven.

It depends on the purpose of running load test on direct runner. As a
runner for testing, direct runner might not need load testing. However, if
running load test on direct runner is used for verifying if load testing
work, then reducing the size of test data definitely works.

-Rui

On Thu, Jan 10, 2019 at 10:10 AM Reuven Lax  wrote:

> The Direct Runner as currently implemented is purposely inefficient. It
> was designed for testing, and therefore does many things that are meant to
> expose bugs in user pipelines (e.g. randomly sorting PCollections,
> serializing/deserializing every element, etc.). So it's not surprising that
> it doesn't behave well under load tests.
>
> Reuven
>
> On Thu, Jan 10, 2019 at 5:55 AM Katarzyna Kucharczyk <
> ka.kucharc...@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> My name is Kasia and I contribute to Beam's tests. Currently, I am
>> working with Łukasz Gajowy on load tests and we created Jenkins
>> configuration to run Synthetic Sources test on DirectRunner. It was decided
>> to generate 1 000 000 000 records (bytes) for a small suite (details you
>> can find in this proposal [1] ). Running this test on the Beam’s Jenkins is
>> causing runtime exception with the message:
>> "java.lang.OutOfMemoryError: GC overhead limit exceeded".
>>
>> Of course, this is not a surprise since it's a lot of data. That's why I
>> am asking for your advice/opinion:
>> Do you think if this test should be smaller? On the other hand, if it's
>> going to be smaller would it be still worth testing as a load test?
>> Maybe it would be better to wait for the UniversalLocalRunner instead and
>> use it while it's there? What is the status of ULR?  Do you know if the ULR
>> will replace DirectRunner?
>>
>> I created an issue [2] with details of this problem where you can find
>> the link to the example of a failing job.
>>
>> Thanks,
>> Kasia
>>
>> [1] https://s.apache.org/load-test-basic-operations
>> 
>> [2] https://issues.apache.org/jira/browse/BEAM-6351
>>
>

Re: Load testing on DirectRunner

[Reuven Lax]

The Direct Runner as currently implemented is purposely inefficient. It was
designed for testing, and therefore does many things that are meant to
expose bugs in user pipelines (e.g. randomly sorting PCollections,
serializing/deserializing every element, etc.). So it's not surprising that
it doesn't behave well under load tests.

Reuven

On Thu, Jan 10, 2019 at 5:55 AM Katarzyna Kucharczyk <
ka.kucharc...@gmail.com> wrote:

> Hi Everyone,
>
> My name is Kasia and I contribute to Beam's tests. Currently, I am working
> with Łukasz Gajowy on load tests and we created Jenkins configuration to
> run Synthetic Sources test on DirectRunner. It was decided to generate 1
> 000 000 000 records (bytes) for a small suite (details you can find in this
> proposal [1] ). Running this test on the Beam’s Jenkins is causing runtime
> exception with the message:
> "java.lang.OutOfMemoryError: GC overhead limit exceeded".
>
> Of course, this is not a surprise since it's a lot of data. That's why I
> am asking for your advice/opinion:
> Do you think if this test should be smaller? On the other hand, if it's
> going to be smaller would it be still worth testing as a load test?
> Maybe it would be better to wait for the UniversalLocalRunner instead and
> use it while it's there? What is the status of ULR?  Do you know if the ULR
> will replace DirectRunner?
>
> I created an issue [2] with details of this problem where you can find the
> link to the example of a failing job.
>
> Thanks,
> Kasia
>
> [1] https://s.apache.org/load-test-basic-operations
> 
> [2] https://issues.apache.org/jira/browse/BEAM-6351
>

Re: Dev contact for KafkaIO

[Ismaël Mejía]

Hello,

Thanks a lot Raghu for all the nice work you did for KafkaIO and the
project in general.
I hope we can bother you a bit (only when needed) with advanced Kafka
related issues.

Best wishes for your new adventure.
Ismaël

On Thu, Jan 10, 2019 at 11:58 AM Alexey Romanenko
 wrote:
>
> Raghu,
>
> Thank you for your confidence, I’ll be happy to lead a development of KafkaIO 
> as much as I can if, of course, Beam community will be agree with that.
>
> I've been involved in KafkaIO development for some time and it was always a 
> pleasure to work with you on bug fixes and new features.
> Anyway, I don’t say “goodbye" since I’m pretty sure we will discuss together 
> new KafkaIO things and design in the future.
>
> All the best to you with new adventure!
> Alexey
>
> > On 9 Jan 2019, at 20:08, Raghu Angadi  wrote:
> >
> > Hi Everyone,
> >
> > Last Friday was my last day at Google and Dataflow team. I had a great time 
> > and learnt lot from working on both Apache Beam and Dataflow.
> >
> > My new job at a startup is not directly related to Apache Beam and I will 
> > not be able to spend a lot of time on support or development of KafkaIO.
> >
> > Alexey Romanenko has been quite active in Apache Beam community and I had 
> > many productive interactions with him on KafkaIO pull requests. I think he 
> > will be a great lead to take KafkaIO forward. Many of you made contributors 
> > to KafkaIO and I am sure you will continue to do so.
> >
> > I have been a contributor for Apache projects for a long, and I do think it 
> > is important for us to support end users as best as we can. I will be 
> > following Apache Beam developments and will be available for questions and 
> > reviews of pull requests or feature discussions.
> >
> > Thanks,
> > Raghu.
>

Re: error with DirectRunner

[Allie Chen]

Thank you so much for start working on this!

On Thu, Jan 10, 2019 at 5:55 AM Robert Bradshaw  wrote:

> https://github.com/apache/beam/pull/7456
>
> On Thu, Jan 10, 2019 at 10:59 AM Robert Bradshaw 
> wrote:
> >
> > Sorry this got lost. I filed
> > https://issues.apache.org/jira/browse/BEAM-6404; hopefully it'll be an
> > easy fix.
> >
> > On Wed, Jan 9, 2019 at 8:33 PM Allie Chen  wrote:
> > >
> > > Greetings!
> > >
> > > May I ask whether there is any plan to work on this issue? Or if I
> just use `BundleBasedDirectRunner` instead of `DirectRunner`, will there be
> any performance issues/caveats I should worry about?
> > >
> > > Thanks!
> > > Allie
> > >
> > > On Tue, Oct 30, 2018 at 8:13 PM Udi Meiri  wrote:
> > >>
> > >> +Robert Bradshaw I would be happy to debug and fix this, but I'd need
> more guidance on where to look.
> > >>
> > >> On Tue, Oct 30, 2018 at 4:07 PM Udi Meiri  wrote:
> > >>>
> > >>> Created https://issues.apache.org/jira/browse/BEAM-5927
> > >>>
> > >>> On Tue, Oct 30, 2018 at 1:13 PM Lukasz Cwik 
> wrote:
> > 
> >  Udi, do you know if we have a bug tracking this issue?
> > 
> >  If not, can you file one referencing this e-mail thread?
> > 
> >  On Tue, Oct 30, 2018 at 6:33 AM Allie Chen 
> wrote:
> > >
> > > Thanks Udi. I agree, since it works fine removing either the side
> input or the last flatten and combine operation.
> > >
> > > On Mon, Oct 29, 2018 at 9:02 PM Udi Meiri 
> wrote:
> > >>
> > >> This looks like a FnApiRunner bug.
> > >> When I override use_fnapi_runner = False in direct_runner.py the
> pipeline works.
> > >>
> > >> It seems like either the side-input to _copy_number or the
> Flatten operation is the culprit.
> > >>
> > >> On Mon, Oct 29, 2018 at 2:37 PM Allie Chen 
> wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> I have a project that started failing with DirectRunner, but
> works well using DataflowRunner (last working version is 2.4). The error
> message I received are:
> > >>> line 1088, in run_stage
> > >>>   pipeline_components.pcollections[actual_pcoll_id].coder_id]]
> > >>> KeyError: u'ref_Coder_WindowedValueCoder_1'
> > >>>
> > >>> I have simplified the pipeline to the following example. Can
> someone please take a look? Many thanks!
> > >>>
> > >>> Allie
> > >>>
> > >>>
> > >>> import apache_beam as beam
> > >>> import argparse
> > >>> from apache_beam import transforms
> > >>> from apache_beam import pvalue
> > >>> from apache_beam.options import pipeline_options
> > >>>
> > >>>
> > >>> def _copy_number(number, side=None):
> > >>>   yield number
> > >>>
> > >>>
> > >>> def fn_sum(values):
> > >>>   return sum(values)
> > >>>
> > >>>
> > >>> def run(argv=None):
> > >>>   parser = argparse.ArgumentParser()
> > >>>   _, pipeline_args = parser.parse_known_args(argv)
> > >>>   options = pipeline_options.PipelineOptions(pipeline_args)
> > >>>   numbers = [1, 2]
> > >>>   with beam.Pipeline(options=options) as p:
> > >>> sum_1 = (p
> > >>>  | 'ReadNumber1' >> transforms.Create(numbers)
> > >>>  | 'CalculateSum1' >> beam.CombineGlobally(fn_sum))
> > >>>
> > >>> sum_2 = (p
> > >>>  | 'ReadNumber2' >> transforms.Create(numbers)
> > >>>  | beam.ParDo(_copy_number,
> pvalue.AsSingleton(sum_1))
> > >>>  | 'CalculateSum2' >> beam.CombineGlobally(fn_sum))
> > >>>
> > >>> _ = ((sum_1, sum_2)
> > >>>  | beam.Flatten()
> > >>>  | 'CalculateSum3' >> beam.CombineGlobally(fn_sum)
> > >>>  | beam.io.WriteToText('gs://BUCKET/sum'))
> > >>>
> > >>>
> > >>>
>

Load testing on DirectRunner

[Katarzyna Kucharczyk]

Hi Everyone,

My name is Kasia and I contribute to Beam's tests. Currently, I am working
with Łukasz Gajowy on load tests and we created Jenkins configuration to
run Synthetic Sources test on DirectRunner. It was decided to generate 1
000 000 000 records (bytes) for a small suite (details you can find in this
proposal [1] ). Running this test on the Beam’s Jenkins is causing runtime
exception with the message:
"java.lang.OutOfMemoryError: GC overhead limit exceeded".

Of course, this is not a surprise since it's a lot of data. That's why I am
asking for your advice/opinion:
Do you think if this test should be smaller? On the other hand, if it's
going to be smaller would it be still worth testing as a load test?
Maybe it would be better to wait for the UniversalLocalRunner instead and
use it while it's there? What is the status of ULR?  Do you know if the ULR
will replace DirectRunner?

I created an issue [2] with details of this problem where you can find the
link to the example of a failing job.

Thanks,
Kasia

[1] https://s.apache.org/load-test-basic-operations

[2] https://issues.apache.org/jira/browse/BEAM-6351

Re: Dev contact for KafkaIO

[Alexey Romanenko]

Raghu,

Thank you for your confidence, I’ll be happy to lead a development of KafkaIO 
as much as I can if, of course, Beam community will be agree with that.

I've been involved in KafkaIO development for some time and it was always a 
pleasure to work with you on bug fixes and new features.
Anyway, I don’t say “goodbye" since I’m pretty sure we will discuss together 
new KafkaIO things and design in the future. 

All the best to you with new adventure!
Alexey 

> On 9 Jan 2019, at 20:08, Raghu Angadi  wrote:
> 
> Hi Everyone,
> 
> Last Friday was my last day at Google and Dataflow team. I had a great time 
> and learnt lot from working on both Apache Beam and Dataflow.
> 
> My new job at a startup is not directly related to Apache Beam and I will not 
> be able to spend a lot of time on support or development of KafkaIO.
> 
> Alexey Romanenko has been quite active in Apache Beam community and I had 
> many productive interactions with him on KafkaIO pull requests. I think he 
> will be a great lead to take KafkaIO forward. Many of you made contributors 
> to KafkaIO and I am sure you will continue to do so.
> 
> I have been a contributor for Apache projects for a long, and I do think it 
> is important for us to support end users as best as we can. I will be 
> following Apache Beam developments and will be available for questions and 
> reviews of pull requests or feature discussions.
> 
> Thanks,
> Raghu.

Re: error with DirectRunner

[Robert Bradshaw]

https://github.com/apache/beam/pull/7456

On Thu, Jan 10, 2019 at 10:59 AM Robert Bradshaw  wrote:
>
> Sorry this got lost. I filed
> https://issues.apache.org/jira/browse/BEAM-6404; hopefully it'll be an
> easy fix.
>
> On Wed, Jan 9, 2019 at 8:33 PM Allie Chen  wrote:
> >
> > Greetings!
> >
> > May I ask whether there is any plan to work on this issue? Or if I just use 
> > `BundleBasedDirectRunner` instead of `DirectRunner`, will there be any 
> > performance issues/caveats I should worry about?
> >
> > Thanks!
> > Allie
> >
> > On Tue, Oct 30, 2018 at 8:13 PM Udi Meiri  wrote:
> >>
> >> +Robert Bradshaw I would be happy to debug and fix this, but I'd need more 
> >> guidance on where to look.
> >>
> >> On Tue, Oct 30, 2018 at 4:07 PM Udi Meiri  wrote:
> >>>
> >>> Created https://issues.apache.org/jira/browse/BEAM-5927
> >>>
> >>> On Tue, Oct 30, 2018 at 1:13 PM Lukasz Cwik  wrote:
> 
>  Udi, do you know if we have a bug tracking this issue?
> 
>  If not, can you file one referencing this e-mail thread?
> 
>  On Tue, Oct 30, 2018 at 6:33 AM Allie Chen  wrote:
> >
> > Thanks Udi. I agree, since it works fine removing either the side input 
> > or the last flatten and combine operation.
> >
> > On Mon, Oct 29, 2018 at 9:02 PM Udi Meiri  wrote:
> >>
> >> This looks like a FnApiRunner bug.
> >> When I override use_fnapi_runner = False in direct_runner.py the 
> >> pipeline works.
> >>
> >> It seems like either the side-input to _copy_number or the Flatten 
> >> operation is the culprit.
> >>
> >> On Mon, Oct 29, 2018 at 2:37 PM Allie Chen  
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I have a project that started failing with DirectRunner, but works 
> >>> well using DataflowRunner (last working version is 2.4). The error 
> >>> message I received are:
> >>> line 1088, in run_stage
> >>>   pipeline_components.pcollections[actual_pcoll_id].coder_id]]
> >>> KeyError: u'ref_Coder_WindowedValueCoder_1'
> >>>
> >>> I have simplified the pipeline to the following example. Can someone 
> >>> please take a look? Many thanks!
> >>>
> >>> Allie
> >>>
> >>>
> >>> import apache_beam as beam
> >>> import argparse
> >>> from apache_beam import transforms
> >>> from apache_beam import pvalue
> >>> from apache_beam.options import pipeline_options
> >>>
> >>>
> >>> def _copy_number(number, side=None):
> >>>   yield number
> >>>
> >>>
> >>> def fn_sum(values):
> >>>   return sum(values)
> >>>
> >>>
> >>> def run(argv=None):
> >>>   parser = argparse.ArgumentParser()
> >>>   _, pipeline_args = parser.parse_known_args(argv)
> >>>   options = pipeline_options.PipelineOptions(pipeline_args)
> >>>   numbers = [1, 2]
> >>>   with beam.Pipeline(options=options) as p:
> >>> sum_1 = (p
> >>>  | 'ReadNumber1' >> transforms.Create(numbers)
> >>>  | 'CalculateSum1' >> beam.CombineGlobally(fn_sum))
> >>>
> >>> sum_2 = (p
> >>>  | 'ReadNumber2' >> transforms.Create(numbers)
> >>>  | beam.ParDo(_copy_number, pvalue.AsSingleton(sum_1))
> >>>  | 'CalculateSum2' >> beam.CombineGlobally(fn_sum))
> >>>
> >>> _ = ((sum_1, sum_2)
> >>>  | beam.Flatten()
> >>>  | 'CalculateSum3' >> beam.CombineGlobally(fn_sum)
> >>>  | beam.io.WriteToText('gs://BUCKET/sum'))
> >>>
> >>>
> >>>

Re: error with DirectRunner

[Robert Bradshaw]

Sorry this got lost. I filed
https://issues.apache.org/jira/browse/BEAM-6404; hopefully it'll be an
easy fix.

On Wed, Jan 9, 2019 at 8:33 PM Allie Chen  wrote:
>
> Greetings!
>
> May I ask whether there is any plan to work on this issue? Or if I just use 
> `BundleBasedDirectRunner` instead of `DirectRunner`, will there be any 
> performance issues/caveats I should worry about?
>
> Thanks!
> Allie
>
> On Tue, Oct 30, 2018 at 8:13 PM Udi Meiri  wrote:
>>
>> +Robert Bradshaw I would be happy to debug and fix this, but I'd need more 
>> guidance on where to look.
>>
>> On Tue, Oct 30, 2018 at 4:07 PM Udi Meiri  wrote:
>>>
>>> Created https://issues.apache.org/jira/browse/BEAM-5927
>>>
>>> On Tue, Oct 30, 2018 at 1:13 PM Lukasz Cwik  wrote:

 Udi, do you know if we have a bug tracking this issue?

 If not, can you file one referencing this e-mail thread?

 On Tue, Oct 30, 2018 at 6:33 AM Allie Chen  wrote:
>
> Thanks Udi. I agree, since it works fine removing either the side input 
> or the last flatten and combine operation.
>
> On Mon, Oct 29, 2018 at 9:02 PM Udi Meiri  wrote:
>>
>> This looks like a FnApiRunner bug.
>> When I override use_fnapi_runner = False in direct_runner.py the 
>> pipeline works.
>>
>> It seems like either the side-input to _copy_number or the Flatten 
>> operation is the culprit.
>>
>> On Mon, Oct 29, 2018 at 2:37 PM Allie Chen  wrote:
>>>
>>> Hi,
>>>
>>> I have a project that started failing with DirectRunner, but works well 
>>> using DataflowRunner (last working version is 2.4). The error message I 
>>> received are:
>>> line 1088, in run_stage
>>>   pipeline_components.pcollections[actual_pcoll_id].coder_id]]
>>> KeyError: u'ref_Coder_WindowedValueCoder_1'
>>>
>>> I have simplified the pipeline to the following example. Can someone 
>>> please take a look? Many thanks!
>>>
>>> Allie
>>>
>>>
>>> import apache_beam as beam
>>> import argparse
>>> from apache_beam import transforms
>>> from apache_beam import pvalue
>>> from apache_beam.options import pipeline_options
>>>
>>>
>>> def _copy_number(number, side=None):
>>>   yield number
>>>
>>>
>>> def fn_sum(values):
>>>   return sum(values)
>>>
>>>
>>> def run(argv=None):
>>>   parser = argparse.ArgumentParser()
>>>   _, pipeline_args = parser.parse_known_args(argv)
>>>   options = pipeline_options.PipelineOptions(pipeline_args)
>>>   numbers = [1, 2]
>>>   with beam.Pipeline(options=options) as p:
>>> sum_1 = (p
>>>  | 'ReadNumber1' >> transforms.Create(numbers)
>>>  | 'CalculateSum1' >> beam.CombineGlobally(fn_sum))
>>>
>>> sum_2 = (p
>>>  | 'ReadNumber2' >> transforms.Create(numbers)
>>>  | beam.ParDo(_copy_number, pvalue.AsSingleton(sum_1))
>>>  | 'CalculateSum2' >> beam.CombineGlobally(fn_sum))
>>>
>>> _ = ((sum_1, sum_2)
>>>  | beam.Flatten()
>>>  | 'CalculateSum3' >> beam.CombineGlobally(fn_sum)
>>>  | beam.io.WriteToText('gs://BUCKET/sum'))
>>>
>>>
>>>

Re: [DISCUSS] (Forked thread) Beam issue triage & assignees

[Mikhail Gryzykhin]

+1 to keep issues unassigned and reevaluate backlog from time to time.

We can also auto-unassign if there was no activity on ticket for N days. Or
we can have auto-mailed report that highlights stale assigned issues.

On Thu, Jan 10, 2019 at 12:10 AM Robert Bradshaw 
wrote:

> On Thu, Jan 10, 2019 at 3:20 AM Ahmet Altay  wrote:
> >
> > I agree with the proposals here. Initial state of "Needs Review" and
> blocking releases on untriaged issues will ensure that we will at least
> look at every new issue once.
>
> +1.
>
> I'm more ambivalent about closing stale issues. Unlike PRs, issues can
> be filed as "we should (not forget to) do this" much sooner than
> they're actively worked on.
>
> > On Wed, Jan 9, 2019 at 10:30 AM Maximilian Michels 
> wrote:
> >>
> >> Hi Kenn,
> >>
> >> As your data shows, default-assigning issues to a single person does not
> >> automatically solve triaging issues. Quite the contrary, it hides the
> triage
> >> status of an issue.
> >>
> >>  From the perspective of the Flink Runner, we used to auto-assign but
> we got rid
> >> of this. Instead, we monitor the newly coming issues and take actions.
> We also
> >> go through the old ones occasionally. I believe that works fine for us.
> >>
> >> The Flink project itself also does not default-assign, newly created
> issues are
> >> unassigned. There are component leads overseeing issues. There is no
> guarantee
> >> that every issue gets triaged.
> >>
> >> "Needs Triage" or or "Needs Review" (seems easier to understand of
> non-native
> >> speakers) sounds like a good addition, but it will not solve the
> problem that
> >> issues need to be curated and maintained after the initial triage. For
> example,
> >> I've seen issues not closed after they have been fixed via a PR.
> However, "Needs
> >> Triage" will ensure that all issues get looked at. This could be
> helpful for
> >> releases, if not-yet-triaged issues are looked at early enough.
> >>
> >> I'd suggest to:
> >>
> >> 1) Change new issues to be Unassigned and to be in status "Needs Review"
> >> 2) Remove Assignees from all not-being-worked-on issues
> >
> >
> > For the existing issues, I suggest unassign all issues assigned to
> people with > N issues for a large N. Something like 30, > %1 of all
> issues. There are also issues assigned to people who are no longer active
> in the community. We could unassign those as well.
> >
> > Another issue is average age for open issues is also ever growing and is
> over > 300 days now. It would be nice if we can have an equivalent of stale
> bot for PRs for JIRAs, unassigning/closing stale issues periodically.
> >
> >>
> >> 3) Ensure that each component's unresolved issues get looked at
> regularly
> >>
> >> I suppose how to do 3) effectively is an open question. I currently
> don't see a
> >> better way as to oversee each component by multiple committers, similar
> to the
> >> OWNERS idea that we once had. I think we should move the OWNERS
> information to a
> >> Wiki page to make ownership/maintainership more transparent and easier
> to change.
> >
> >
> > I think this is a good idea. We can also divide components even more and
> there will be more people looking at smaller components each. We could also
> consider defining SLO type metrics especially for high priority issues, and
> present those in a dashboard. That way we can collectively see the overall
> health of our triaging efforts and prevent important issues from being
> forgotten.
> >
> >>
> >>
> >> Thanks,
> >> Max
> >>
> >> On 08.01.19 16:30, Kenneth Knowles wrote:
> >> > Forking discussion on
> >> >
> >> >   - Some folks have 100+ issues assigned
> >> >   - Jira components might not be the best way to get things triaged
> >> >
> >> > I just ran a built-in Jira report to summarize how many tickets are
> claimed by
> >> > different users [1]. It does look like component owners tend to
> accumulate
> >> > issues and they are not likely to get addressed.
> >> >
> >> > So what do we want and how can we make it happen? Here is what I
> want, personally:
> >> >
> >> >   - every new issue gets looked at by someone who can decide the right
> >> > component, who to ping, etc
> >> >   - no person is assigned an issue that they are not active on
> >> >   - we don't release with potential bugs waiting to be triaged
> >> >
> >> > The current status it that issues get assigned to a component owner
> when filed.
> >> > The component owner should route it and it most likely will end up in
> some
> >> > component unassigned.
> >> >
> >> > Another possibility is that we add a "Needs Triage" status and figure
> out a way
> >> > to make sure they are all looked at - maybe also by blocking a
> release on having
> >> > zero Needs Triage bugs. Just an idea.
> >> >
> >> > And important question I did not look into: what do other Apache or
> Apache-style
> >> > decentralized projects do?
> >> >
> >> > Kenn
> >> >
> >> > [1]
> >> >
> https://issues.apache.org/jira/secure/ConfigureReport.jspa?projectO

Re: [DISCUSS] (Forked thread) Beam issue triage & assignees

[Robert Bradshaw]

On Thu, Jan 10, 2019 at 3:20 AM Ahmet Altay  wrote:
>
> I agree with the proposals here. Initial state of "Needs Review" and blocking 
> releases on untriaged issues will ensure that we will at least look at every 
> new issue once.

+1.

I'm more ambivalent about closing stale issues. Unlike PRs, issues can
be filed as "we should (not forget to) do this" much sooner than
they're actively worked on.

> On Wed, Jan 9, 2019 at 10:30 AM Maximilian Michels  wrote:
>>
>> Hi Kenn,
>>
>> As your data shows, default-assigning issues to a single person does not
>> automatically solve triaging issues. Quite the contrary, it hides the triage
>> status of an issue.
>>
>>  From the perspective of the Flink Runner, we used to auto-assign but we got 
>> rid
>> of this. Instead, we monitor the newly coming issues and take actions. We 
>> also
>> go through the old ones occasionally. I believe that works fine for us.
>>
>> The Flink project itself also does not default-assign, newly created issues 
>> are
>> unassigned. There are component leads overseeing issues. There is no 
>> guarantee
>> that every issue gets triaged.
>>
>> "Needs Triage" or or "Needs Review" (seems easier to understand of non-native
>> speakers) sounds like a good addition, but it will not solve the problem that
>> issues need to be curated and maintained after the initial triage. For 
>> example,
>> I've seen issues not closed after they have been fixed via a PR. However, 
>> "Needs
>> Triage" will ensure that all issues get looked at. This could be helpful for
>> releases, if not-yet-triaged issues are looked at early enough.
>>
>> I'd suggest to:
>>
>> 1) Change new issues to be Unassigned and to be in status "Needs Review"
>> 2) Remove Assignees from all not-being-worked-on issues
>
>
> For the existing issues, I suggest unassign all issues assigned to people 
> with > N issues for a large N. Something like 30, > %1 of all issues. There 
> are also issues assigned to people who are no longer active in the community. 
> We could unassign those as well.
>
> Another issue is average age for open issues is also ever growing and is over 
> > 300 days now. It would be nice if we can have an equivalent of stale bot 
> for PRs for JIRAs, unassigning/closing stale issues periodically.
>
>>
>> 3) Ensure that each component's unresolved issues get looked at regularly
>>
>> I suppose how to do 3) effectively is an open question. I currently don't 
>> see a
>> better way as to oversee each component by multiple committers, similar to 
>> the
>> OWNERS idea that we once had. I think we should move the OWNERS information 
>> to a
>> Wiki page to make ownership/maintainership more transparent and easier to 
>> change.
>
>
> I think this is a good idea. We can also divide components even more and 
> there will be more people looking at smaller components each. We could also 
> consider defining SLO type metrics especially for high priority issues, and 
> present those in a dashboard. That way we can collectively see the overall 
> health of our triaging efforts and prevent important issues from being 
> forgotten.
>
>>
>>
>> Thanks,
>> Max
>>
>> On 08.01.19 16:30, Kenneth Knowles wrote:
>> > Forking discussion on
>> >
>> >   - Some folks have 100+ issues assigned
>> >   - Jira components might not be the best way to get things triaged
>> >
>> > I just ran a built-in Jira report to summarize how many tickets are 
>> > claimed by
>> > different users [1]. It does look like component owners tend to accumulate
>> > issues and they are not likely to get addressed.
>> >
>> > So what do we want and how can we make it happen? Here is what I want, 
>> > personally:
>> >
>> >   - every new issue gets looked at by someone who can decide the right
>> > component, who to ping, etc
>> >   - no person is assigned an issue that they are not active on
>> >   - we don't release with potential bugs waiting to be triaged
>> >
>> > The current status it that issues get assigned to a component owner when 
>> > filed.
>> > The component owner should route it and it most likely will end up in some
>> > component unassigned.
>> >
>> > Another possibility is that we add a "Needs Triage" status and figure out 
>> > a way
>> > to make sure they are all looked at - maybe also by blocking a release on 
>> > having
>> > zero Needs Triage bugs. Just an idea.
>> >
>> > And important question I did not look into: what do other Apache or 
>> > Apache-style
>> > decentralized projects do?
>> >
>> > Kenn
>> >
>> > [1]
>> > https://issues.apache.org/jira/secure/ConfigureReport.jspa?projectOrFilterId=filter-12343976&statistictype=assignees&selectedProjectId=12319527&reportKey=com.atlassian.jira.jira-core-reports-plugin%3Apie-report&atl_token=A5KQ-2QAV-T4JA-FDED%7C4ba43752834c87065a99ddea7c1ea92342aa%7Clin&Next=Next