date:20190314

Honestly, I dod not even consider Runner and Beam FileSystem dependencies
here, which will most likely also add some complexity to the problem here.
And there might be more.

On Thu, Mar 14, 2019 at 8:53 PM Kenneth Knowles  wrote:

> Thanks for the super-detailed explanation and illustration.
>
> I especially think you emphasized something we generally have not
> considered: the end-user pinning a different version of a client lib, and
> the fact that if we vendor the client lib they cannot change the version.
> To me, that makes vendoring client libs kind of a deal-breaker. So I think:
>
> a) yes, solved by vendored (already done)
> b) leaving the IO's version of Guava on the classpath seems the only
> solution, and end-user has to resolve
> c) good point that in the not-on-API-surface case also, we can't really
> tweak anything and we need the end-user to resolve
>
> So the main action item is to remove the top-level Guava version and to
> let IOs determine their Guava version, with resolution of diamond deps when
> needed in test configurations.
>
> Kenn
>
> On Wed, Mar 6, 2019 at 3:18 AM Michael Luckey  wrote:
>
>> Unfortunately, I am unable to see, how option 2 could be done from beam
>> side? Or is it meant to delay to the end user?
>>
>> Let me try to elaborate:
>>
>> Assume user implements a pipeline with 2 different IOs, both depending on
>> incompatible versions of e.g. guava. Also, user code has a (direct or
>> transitive) dependency to
>> yet another incompatible version of guava. I ll try to get into more
>> details later, but for now assume
>>
>>  AIO    ParDoParDo    Pardo   BIO
>>|  |
>> |
>> guava-X.jar   guava-Y.jar
>>  guava-Z.jar
>>
>> What currently happens (AFAIU) is during build we do pin the versions of
>> guava to some fixed version [1], which means AIO and BIO are compiled with
>> guava-20, which might or might not
>> be ABI compatible to guava-X and/or guava-Z. (This is the problem these 2
>> users encountered, right?)
>>
>> Now the user trying to run this pipeline has to decide, which version to
>> use. Whatever she chooses the might be incompatibilities, She could do so
>> with maven, and of course also with Gradle or any other build system
>> (hopefully).
>>
>> If we now replace guava.20 with the newest on build, lets say
>> guava-LATEST which happens to be a dependency of CIO, how would that change
>> the game. So is there anything beam could do about it?
>>
>> Now I try to understand those deps in more detail. We have those IOs, and
>> the user code thingy. I assume, beam code switched to vendors guava, so
>> nothing is exposed here, we only have issues on external interfaces as IOs.
>>
>> 1. The user code.
>> Whether guava is used directly or transitive by some used library
>> dependency, there is nothing beam can do about it. The user has to solve
>> this issue by herself. She will probably encounter exactly the same
>> problems as beam within the IOs,
>>
>> 2. IOs
>>
>> a: Beam IO code uses guava, no transitive deps. This is solved by
>> vendoring, right?
>>
>>SomeIO <--- guava-20.jar   is replaced by
>>  SomeIO <--- vendored_guava_v20.ja
>>
>> b: Beam uses some client lib (e.g. some-client-core) which has a
>> dependency on guava but does not expose any guava class to the outside
>>
>>SomeIO < some-client-lib < guava-X.jar
>>
>>  In this case beam 'could' relocate the client lib and 'solve' that guava
>> exposing on the class path, *but* we might be tight to a specific
>> version of the backend. as the user can not replace the 'some-client-core'
>> dependency with a different version (hoping there are no interop issues
>> with beams IO)
>> I guess, we are better off, not doing that? Which unfortuantely means
>> delaying version resolution to the end user, as transitive dependency will
>> end up on the end user runtime classpath and might cause diamond
>> dependency issues.
>>
>> c: Beam uses some client lib (e.g. bigquery-client-core) which has a
>> dependency on guava which does expose guava classes in its external API
>>
>>SomeIO < some-client-lib < guava-X.jar
>>   |  |
>>   ---
>>
>> We could of course also try to shade here with above drawbacks, but this
>> might not even work as the client lib communicates with the backend and
>> might also expose guava there. So the best we could do is, as Kenn stated,
>> to ensure interoperability SomeIO with the client.
>>
>> So to my current understanding I second Kenn here. We should stop forcing
>> a specific version of (of course uncensored) guava and align on IO basis to
>> the requirements of dependencies. Still, on that lower (IO) level we might
>> need to force some version as already there we might have diamond deps on
>> guava. But I think we are

Re: Python sdist broken?

Thanks Katarzyna for that explanation.

On Thu, Mar 14, 2019 at 8:37 PM Katarzyna Kucharczyk <
ka.kucharc...@gmail.com> wrote:

> Hi,
> Michael - many thanks for letting know and Ahmet for reverting. I think I
> know which task caused the problem.
>
> A quick explanation about what's happened. Load test gradle task was
> created in load_tests module. A build/ directory which was excluded in
> 'sdist' task wasn't the same as the one created inside the module. While
> copying it this created some kind of loop. I already suggested my solution
> to Mark.
>
> @Valentyn, the load_tests (in smoke option) haven't been added to
> preCommit as not important to check in a different way than on demand (by
> phrase triggering). Anyway, this situation shows that such decision should
> be re-thought. Also, a solution here would be moving load_test task to the
> main gradle python file. I decided to keep them separately because they
> weren't designed to be part of big task such as preCommit.
>

We should keep in mind, that creating a submodule is in fact really a
heavyweight process. Whereas adding a task should be considered pretty
lightweight.IIUC, that load test module only adds a single task, which
might be called with different args. If so, that task is already 'kept
separate' in itself.

So if this kind of separation is the only reason - in contrast to a more
semantic 'this *is* a different subproject' we probably should restrain
from creating as such. But my understanding could be wrong here.

Regarding the miss out with pre commit test, it is unfortunately not enough
to rely on some subset of tasks if doing such heavy lifting on the build
system. Even a full build will not reveal all problems, as gradle only
configures and of course runs what is asked for. There is probably not much
we can do about that apart from executing everything, which seems
unfeasible.


>
> Let me know what you think.
>
> Kasia
>
> On Thu, Mar 14, 2019 at 7:32 PM Michael Luckey 
> wrote:
>
>> Thanks for that revert!
>>
>> I d really love to get those proposed changes of #7675 in, as this
>> '--keep-temp' also causes some troubles during build.
>>
>> Please let me know if/how I could support here.
>>
>> michel
>>
>> On Thu, Mar 14, 2019 at 5:21 PM Ahmet Altay  wrote:
>>
>>> Sent https://github.com/apache/beam/pull/8059 to revert #7675.
>>>
>>> On Thu, Mar 14, 2019 at 8:10 AM Valentyn Tymofieiev 
>>> wrote:
>>>
 This most likely caused by https://github.com/apache/beam/pull/7675. I
 suggest we revert and roll-forward with a fix. We should also understand
 why this didn't surface in the precommit tests.

 On Thu, Mar 14, 2019 at 4:44 AM Michael Luckey 
 wrote:

> Hi,
>
> while trying to get build working on my vanilla vm, I encounter
> strangely failing python sdists tasks. It keeps complaining about being
> unable to copy files. Those files are in a deeply recursive structure 
> which
> led me to the assumption this might not be intended. This seems to be
> caused by merge of [1], not verified though.
>
> This also happens on our nightly snapshot build [2]. Anyone else
> having this problem?
>
> thx,
>
> michel
>
> [1] https://github.com/apache/beam/pull/7675
> [2] https://scans.gradle.com/s/rllnbwj4lj4qc
>
>
>

Re: JIRA hygiene

2019-03-14 Thread Reuven Lax

I have in the past created JIRAs that took several PRs to close. Often this
happened when I decided that a PR was too large and split it up into
several smaller PRs.

On Thu, Mar 14, 2019 at 12:09 PM Thomas Weise  wrote:

> I see many other projects where JIRAs are resolved when the PR is merged.
> It is simple and seems to work well.
>
> As a reviewer that merges a PR, I need to have an understanding of the
> scope of the work. I cannot remember an instance where it wasn't clear to
> me if a JIRA should be resolved or not. Do others have examples?
>
> It is important that JIRAs are resolved and fields like component, issue
> type and fix version are properly set for the releases. IMO the committer
> is in the best position to verify that.
>
> The contributor usually isn't incentivised to massage a JIRA ticket after
> the code was merged. That is probably the main reason why we find ourselves
> with many dangling tickets. Merge is usually the last action in the
> workflow, at which point we also know what the fix version is.
>
> Thomas
>
>
>
>
>
> On Thu, Mar 14, 2019 at 11:58 AM Mikhail Gryzykhin 
> wrote:
>
>> I believe that there are too many scenarios that we have to cover if we
>> are to design a generic approach. Common pattern I've seen most times is
>> when assignee on the ticket, who's usually author of relevant PR, is
>> expected to either resolve ticket or pass it to the feature owner for
>> verification.
>>
>> We can have a bot that will check stale assigned tickets and poke
>> assignees. Can go further and allow bot to unassign tickets if no response
>> comes and remove "triaged" label. This will always highlight all
>> non-updated tickets and keep forgotten tickets in available pool. Giving a
>> hint to pass ownership of ticket to committer (or person who merged PR) can
>> be a simple answer for contributors who are not sure whether ticket can be
>> closed.
>>
>> --Mikhail
>>
>> Have feedback ?
>>
>>
>> On Wed, Mar 13, 2019 at 6:00 PM Michael Luckey 
>> wrote:
>>
>>> Totally agree. The contributor is most likely be the better target. But
>>> as she is probably less familiar with the process, we might be better of to
>>> put the responsibility on the committer to kindly ask/discuss with her how
>>> to proceed with corresponding jira ticket?
>>>
>>> On Thu, Mar 14, 2019 at 1:18 AM Ahmet Altay  wrote:
>>>
 I agree with defining the workflow for closing JIRAs. Would not
 contributor be in a better position to close JIRAs or keep it open? It
 would make sense for the committer to ask about this but I think
 contributor (presumably the person who is the assignee of the JIRA) could
 be the responsible party for updating their JIRAs. On the other hand, I
 understand the argument that committer could do this at the time of merging
 and fill a gap in the process.

 On Wed, Mar 13, 2019 at 4:59 PM Michael Luckey 
 wrote:

> Hi,
>
> definitely +1 to properly establish a workflow to maintain jira
> status. Naively I d think, the reporter should close as she is the one to
> confirm whether the reported issue is fixed or not. But for obvious 
> reasons
> that will not work here, so - although it puts another burden on
> committers, you are probably right that the committer is the best choice 
> to
> ensure that the ticket gets promoted. Whether it will be resolved or
> clarified what's still to be done.
>
> Looking into the current state, we seem to have tons of issues whith
> merged PRs, which for anyone trying to find an existing jira issue to work
> on makes it unnecessary difficult to decide whether to look into that or
>  not. From my personal experience, it is somehow frustrating going through
> open issues, selecting one and after investing some (or even more) time to
> first understand a problem and then the PR to realise nothing has to be
> done anymore. Or not knowing what's left out and for what reason. But of
> course, this is another issue which we definitely need to invest time into
> - kenn already asked for our support here.
>
> thx,
>
> michel
>
> On Tue, Mar 12, 2019 at 11:30 AM Etienne Chauchot <
> echauc...@apache.org> wrote:
>
>> Hi Thomas,
>>
>> I agree, the committer that merges a PR should close the ticket. And,
>> if needed, he could discuss with the author (inside the PR) to assess if
>> the PR covers the ticket scope.
>>
>> This is the rule I apply to myself when I merge a PR (even thought it
>> has happened that I forgot to close one or two tickets :) ) .
>>
>> Etienne
>>
>>
>> Le lundi 11 mars 2019 à 14:17 -0700, Thomas Weise a écrit :
>>
>> JIRA probably deserves a separate discussion. It is messy.. We also
>> have examples of tickets being referenced by users that were not closed,
>> although the feature long implemented or

Re: Python PVR Reference post-commit tests failing

I know the Java DirectRunner shares a lot of code with the ULR. I'm a bit
unclear on the delta and how independent they are.

Kenn

On Thu, Mar 14, 2019 at 2:10 PM Mikhail Gryzykhin  wrote:

> @Kenneth
> If we disable tests, I'd call Java ULR a dead code.
>
> One of the better compromises:
> 1. disable tests.
> 2. Add tag to the last commit where Java ULR existed.
> 3. Remove Java ULR from head.
>
> Keeping history, no extra dead code at head.
>
> --Mikhail
>
> Have feedback ?
>
>
> On Thu, Mar 14, 2019 at 1:02 PM Ankur Goenka  wrote:
>
>> On that note, we should also think about adding PVR for python reference
>> runners. Jira: https://issues.apache.org/jira/browse/BEAM-6837
>>
>>
>> On Thu, Mar 14, 2019 at 12:57 PM Kenneth Knowles  wrote:
>>
>>> How about this compromise:
>>>
>>> 1. disable the test since clearly no one is relying on the functionality
>>> that is broken
>>> 2. leave the Java ULR as-is for now, and a volunteer can pick it up and
>>> make it work if they want
>>>
>>> Kenn
>>>
>>> On Thu, Mar 14, 2019 at 11:41 AM Mikhail Gryzykhin 
>>> wrote:
>>>
 Hi everyone,

 We have Python PVR Reference post-commit tests failing for quite some
 time now. These are tests for java reference runner.

 According to this thread
 ,
 we are deciding what to do with java reference runner and might want to
 remove it from code base.

 My question is: do we want to a) invest time in fixing python PVR
 tests, or b) disable this test and start cleaning up code?

 a) Is worth it if we want to invest into java reference runner in the
 future.
 b) Is worth if we want to invest into Python and forfeit java reference
 runner.

 Option b) seem more reasonable to me atm, since most people lean
 towards going forward with Python reference runner.

 Please, share your thoughts.

 Regards,
 --Mikhail

 Have feedback ?

>>>

Re: Python PVR Reference post-commit tests failing

@Kenneth
If we disable tests, I'd call Java ULR a dead code.

One of the better compromises:
1. disable tests.
2. Add tag to the last commit where Java ULR existed.
3. Remove Java ULR from head.

Keeping history, no extra dead code at head.

--Mikhail

Have feedback ?


On Thu, Mar 14, 2019 at 1:02 PM Ankur Goenka  wrote:

> On that note, we should also think about adding PVR for python reference
> runners. Jira: https://issues.apache.org/jira/browse/BEAM-6837
>
>
> On Thu, Mar 14, 2019 at 12:57 PM Kenneth Knowles  wrote:
>
>> How about this compromise:
>>
>> 1. disable the test since clearly no one is relying on the functionality
>> that is broken
>> 2. leave the Java ULR as-is for now, and a volunteer can pick it up and
>> make it work if they want
>>
>> Kenn
>>
>> On Thu, Mar 14, 2019 at 11:41 AM Mikhail Gryzykhin 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> We have Python PVR Reference post-commit tests failing for quite some
>>> time now. These are tests for java reference runner.
>>>
>>> According to this thread
>>> ,
>>> we are deciding what to do with java reference runner and might want to
>>> remove it from code base.
>>>
>>> My question is: do we want to a) invest time in fixing python PVR tests,
>>> or b) disable this test and start cleaning up code?
>>>
>>> a) Is worth it if we want to invest into java reference runner in the
>>> future.
>>> b) Is worth if we want to invest into Python and forfeit java reference
>>> runner.
>>>
>>> Option b) seem more reasonable to me atm, since most people lean towards
>>> going forward with Python reference runner.
>>>
>>> Please, share your thoughts.
>>>
>>> Regards,
>>> --Mikhail
>>>
>>> Have feedback ?
>>>
>>

Re: unsubscribe

2019-03-14 Thread Pablo Estrada

I believe you have to use dev-unsubscr...@beam.apache.org.

Best
-P.

On Thu, Mar 14, 2019 at 1:06 PM Milan Das  wrote:

> unsubscribe
>
>>

unsubscribe

2019-03-14 Thread Milan Das

unsubscribe

>

Re: Python PVR Reference post-commit tests failing

2019-03-14 Thread Ankur Goenka

On that note, we should also think about adding PVR for python reference
runners. Jira: https://issues.apache.org/jira/browse/BEAM-6837


On Thu, Mar 14, 2019 at 12:57 PM Kenneth Knowles  wrote:

> How about this compromise:
>
> 1. disable the test since clearly no one is relying on the functionality
> that is broken
> 2. leave the Java ULR as-is for now, and a volunteer can pick it up and
> make it work if they want
>
> Kenn
>
> On Thu, Mar 14, 2019 at 11:41 AM Mikhail Gryzykhin 
> wrote:
>
>> Hi everyone,
>>
>> We have Python PVR Reference post-commit tests failing for quite some
>> time now. These are tests for java reference runner.
>>
>> According to this thread
>> ,
>> we are deciding what to do with java reference runner and might want to
>> remove it from code base.
>>
>> My question is: do we want to a) invest time in fixing python PVR tests,
>> or b) disable this test and start cleaning up code?
>>
>> a) Is worth it if we want to invest into java reference runner in the
>> future.
>> b) Is worth if we want to invest into Python and forfeit java reference
>> runner.
>>
>> Option b) seem more reasonable to me atm, since most people lean towards
>> going forward with Python reference runner.
>>
>> Please, share your thoughts.
>>
>> Regards,
>> --Mikhail
>>
>> Have feedback ?
>>
>

Re: Postcommit kiosk dashboard

Addressed comments:
1. Added precommits.
2. Limited timeframe to 7 days. This removed old jobs from table.
2.1 We keep history of all jobs in separate DB that's used by grafana. Some
of deprecated jobs come from there.

--Mikhail

Have feedback ?


On Thu, Mar 14, 2019 at 12:03 PM Michael Luckey  wrote:

> Very nice!
>
> Two questions though:
> - the links on the left should point somewhere?
> - where are the beam_PostCommit_[Java|GO]_GradleBuild coming from? Cant
> find them on Jenkins...
>
> On Thu, Mar 14, 2019 at 7:20 PM Mikhail Gryzykhin 
> wrote:
>
>> we already have https://s.apache.org/beam-community-metrics
>>
>> --Mikhail
>>
>> Have feedback ?
>>
>>
>> On Thu, Mar 14, 2019 at 11:15 AM Pablo Estrada 
>> wrote:
>>
>>> Woaahhh very fanc... this is great. Thanks so much. Love it. - I
>>> also like the Code Velocity dashboard that you've added.
>>>
>>> Let's make these more discoverable. How about adding a shortlink?
>>> s.apache.org/beam-dash ? : )
>>> Best
>>> -P.
>>>
>>> On Thu, Mar 14, 2019 at 10:58 AM Mikhail Gryzykhin 
>>> wrote:
>>>
 Hi everyone,

 I've added a kiosk style
 
  post-commit
 status dashboard
 
 that can help decorate your office space with green and red colors.

 Regards,
 --Mikhail

 Have feedback ?

>>>

Re: Python PVR Reference post-commit tests failing

How about this compromise:

1. disable the test since clearly no one is relying on the functionality
that is broken
2. leave the Java ULR as-is for now, and a volunteer can pick it up and
make it work if they want

Kenn

On Thu, Mar 14, 2019 at 11:41 AM Mikhail Gryzykhin 
wrote:

> Hi everyone,
>
> We have Python PVR Reference post-commit tests failing for quite some time
> now. These are tests for java reference runner.
>
> According to this thread
> ,
> we are deciding what to do with java reference runner and might want to
> remove it from code base.
>
> My question is: do we want to a) invest time in fixing python PVR tests,
> or b) disable this test and start cleaning up code?
>
> a) Is worth it if we want to invest into java reference runner in the
> future.
> b) Is worth if we want to invest into Python and forfeit java reference
> runner.
>
> Option b) seem more reasonable to me atm, since most people lean towards
> going forward with Python reference runner.
>
> Please, share your thoughts.
>
> Regards,
> --Mikhail
>
> Have feedback ?
>

Re: Merge of vendored Guava (Some PRs need a rebase)

Thanks for the super-detailed explanation and illustration.

I especially think you emphasized something we generally have not
considered: the end-user pinning a different version of a client lib, and
the fact that if we vendor the client lib they cannot change the version.
To me, that makes vendoring client libs kind of a deal-breaker. So I think:

a) yes, solved by vendored (already done)
b) leaving the IO's version of Guava on the classpath seems the only
solution, and end-user has to resolve
c) good point that in the not-on-API-surface case also, we can't really
tweak anything and we need the end-user to resolve

So the main action item is to remove the top-level Guava version and to let
IOs determine their Guava version, with resolution of diamond deps when
needed in test configurations.

Kenn

On Wed, Mar 6, 2019 at 3:18 AM Michael Luckey  wrote:

> Unfortunately, I am unable to see, how option 2 could be done from beam
> side? Or is it meant to delay to the end user?
>
> Let me try to elaborate:
>
> Assume user implements a pipeline with 2 different IOs, both depending on
> incompatible versions of e.g. guava. Also, user code has a (direct or
> transitive) dependency to
> yet another incompatible version of guava. I ll try to get into more
> details later, but for now assume
>
>  AIO    ParDoParDo    Pardo   BIO
>|  |
> |
> guava-X.jar   guava-Y.jar
>  guava-Z.jar
>
> What currently happens (AFAIU) is during build we do pin the versions of
> guava to some fixed version [1], which means AIO and BIO are compiled with
> guava-20, which might or might not
> be ABI compatible to guava-X and/or guava-Z. (This is the problem these 2
> users encountered, right?)
>
> Now the user trying to run this pipeline has to decide, which version to
> use. Whatever she chooses the might be incompatibilities, She could do so
> with maven, and of course also with Gradle or any other build system
> (hopefully).
>
> If we now replace guava.20 with the newest on build, lets say guava-LATEST
> which happens to be a dependency of CIO, how would that change the game. So
> is there anything beam could do about it?
>
> Now I try to understand those deps in more detail. We have those IOs, and
> the user code thingy. I assume, beam code switched to vendors guava, so
> nothing is exposed here, we only have issues on external interfaces as IOs.
>
> 1. The user code.
> Whether guava is used directly or transitive by some used library
> dependency, there is nothing beam can do about it. The user has to solve
> this issue by herself. She will probably encounter exactly the same
> problems as beam within the IOs,
>
> 2. IOs
>
> a: Beam IO code uses guava, no transitive deps. This is solved by
> vendoring, right?
>
>SomeIO <--- guava-20.jar   is replaced by
>  SomeIO <--- vendored_guava_v20.ja
>
> b: Beam uses some client lib (e.g. some-client-core) which has a
> dependency on guava but does not expose any guava class to the outside
>
>SomeIO < some-client-lib < guava-X.jar
>
>  In this case beam 'could' relocate the client lib and 'solve' that guava
> exposing on the class path, *but* we might be tight to a specific version
> of the backend. as the user can not replace the 'some-client-core'
> dependency with a different version (hoping there are no interop issues
> with beams IO)
> I guess, we are better off, not doing that? Which unfortuantely means
> delaying version resolution to the end user, as transitive dependency will
> end up on the end user runtime classpath and might cause diamond
> dependency issues.
>
> c: Beam uses some client lib (e.g. bigquery-client-core) which has a
> dependency on guava which does expose guava classes in its external API
>
>SomeIO < some-client-lib < guava-X.jar
>   |  |
>   ---
>
> We could of course also try to shade here with above drawbacks, but this
> might not even work as the client lib communicates with the backend and
> might also expose guava there. So the best we could do is, as Kenn stated,
> to ensure interoperability SomeIO with the client.
>
> So to my current understanding I second Kenn here. We should stop forcing
> a specific version of (of course uncensored) guava and align on IO basis to
> the requirements of dependencies. Still, on that lower (IO) level we might
> need to force some version as already there we might have diamond deps on
> guava. But I think we are better of not to force a beam global version here.
>
> Of course, that will force the user to handle the problem, but she needs
> to do it anyway, and, unfortunately me also fails to see anything we could
> do better here. (At least current forcing to v20 seems to worsen the
> problem).
>
> [1]
>

Re: Python sdist broken?

2019-03-14 Thread Katarzyna Kucharczyk

Hi,
Michael - many thanks for letting know and Ahmet for reverting. I think I
know which task caused the problem.

A quick explanation about what's happened. Load test gradle task was
created in load_tests module. A build/ directory which was excluded in
'sdist' task wasn't the same as the one created inside the module. While
copying it this created some kind of loop. I already suggested my solution
to Mark.

@Valentyn, the load_tests (in smoke option) haven't been added to preCommit
as not important to check in a different way than on demand (by phrase
triggering). Anyway, this situation shows that such decision should be
re-thought. Also, a solution here would be moving load_test task to the
main gradle python file. I decided to keep them separately because they
weren't designed to be part of big task such as preCommit.

Let me know what you think.

Kasia

On Thu, Mar 14, 2019 at 7:32 PM Michael Luckey  wrote:

> Thanks for that revert!
>
> I d really love to get those proposed changes of #7675 in, as this
> '--keep-temp' also causes some troubles during build.
>
> Please let me know if/how I could support here.
>
> michel
>
> On Thu, Mar 14, 2019 at 5:21 PM Ahmet Altay  wrote:
>
>> Sent https://github.com/apache/beam/pull/8059 to revert #7675.
>>
>> On Thu, Mar 14, 2019 at 8:10 AM Valentyn Tymofieiev 
>> wrote:
>>
>>> This most likely caused by https://github.com/apache/beam/pull/7675. I
>>> suggest we revert and roll-forward with a fix. We should also understand
>>> why this didn't surface in the precommit tests.
>>>
>>> On Thu, Mar 14, 2019 at 4:44 AM Michael Luckey 
>>> wrote:
>>>
 Hi,

 while trying to get build working on my vanilla vm, I encounter
 strangely failing python sdists tasks. It keeps complaining about being
 unable to copy files. Those files are in a deeply recursive structure which
 led me to the assumption this might not be intended. This seems to be
 caused by merge of [1], not verified though.

 This also happens on our nightly snapshot build [2]. Anyone else having
 this problem?

 thx,

 michel

 [1] https://github.com/apache/beam/pull/7675
 [2] https://scans.gradle.com/s/rllnbwj4lj4qc

>>>

Re: Postcommit kiosk dashboard

@Kenneth
Good point.

That's possible and I thought of it. However then total list of jobs will
overflow the size of most screens, so it will not work as kiosk any more.
Our tests jobs list is way too long.

Also, I would not worry about those as much, since people get aware of
pre-commits failures much faster via failing pre-commits on their PRs.

--Mikhail

Have feedback ?

On Thu, Mar 14, 2019 at 11:20 AM Kenneth Knowles  wrote:

> This is great!
>
> "PreCommit_Cron" are actually very important postcommit runs. Can you add
> them?
>
> Kenn
>
> On Thu, Mar 14, 2019 at 11:15 AM Pablo Estrada  wrote:
>
>> Woaahhh very fanc... this is great. Thanks so much. Love it. - I also
>> like the Code Velocity dashboard that you've added.
>>
>> Let's make these more discoverable. How about adding a shortlink?
>> s.apache.org/beam-dash ? : )
>> Best
>> -P.
>>
>> On Thu, Mar 14, 2019 at 10:58 AM Mikhail Gryzykhin 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I've added a kiosk style
>>> 
>>>  post-commit
>>> status dashboard
>>> 
>>> that can help decorate your office space with green and red colors.
>>>
>>> Regards,
>>> --Mikhail
>>>
>>> Have feedback ?
>>>
>>

Re: JIRA hygiene

2019-03-14 Thread Thomas Weise

I see many other projects where JIRAs are resolved when the PR is merged.
It is simple and seems to work well.

As a reviewer that merges a PR, I need to have an understanding of the
scope of the work. I cannot remember an instance where it wasn't clear to
me if a JIRA should be resolved or not. Do others have examples?

It is important that JIRAs are resolved and fields like component, issue
type and fix version are properly set for the releases. IMO the committer
is in the best position to verify that.

The contributor usually isn't incentivised to massage a JIRA ticket after
the code was merged. That is probably the main reason why we find ourselves
with many dangling tickets. Merge is usually the last action in the
workflow, at which point we also know what the fix version is.

Thomas





On Thu, Mar 14, 2019 at 11:58 AM Mikhail Gryzykhin 
wrote:

> I believe that there are too many scenarios that we have to cover if we
> are to design a generic approach. Common pattern I've seen most times is
> when assignee on the ticket, who's usually author of relevant PR, is
> expected to either resolve ticket or pass it to the feature owner for
> verification.
>
> We can have a bot that will check stale assigned tickets and poke
> assignees. Can go further and allow bot to unassign tickets if no response
> comes and remove "triaged" label. This will always highlight all
> non-updated tickets and keep forgotten tickets in available pool. Giving a
> hint to pass ownership of ticket to committer (or person who merged PR) can
> be a simple answer for contributors who are not sure whether ticket can be
> closed.
>
> --Mikhail
>
> Have feedback ?
>
>
> On Wed, Mar 13, 2019 at 6:00 PM Michael Luckey 
> wrote:
>
>> Totally agree. The contributor is most likely be the better target. But
>> as she is probably less familiar with the process, we might be better of to
>> put the responsibility on the committer to kindly ask/discuss with her how
>> to proceed with corresponding jira ticket?
>>
>> On Thu, Mar 14, 2019 at 1:18 AM Ahmet Altay  wrote:
>>
>>> I agree with defining the workflow for closing JIRAs. Would not
>>> contributor be in a better position to close JIRAs or keep it open? It
>>> would make sense for the committer to ask about this but I think
>>> contributor (presumably the person who is the assignee of the JIRA) could
>>> be the responsible party for updating their JIRAs. On the other hand, I
>>> understand the argument that committer could do this at the time of merging
>>> and fill a gap in the process.
>>>
>>> On Wed, Mar 13, 2019 at 4:59 PM Michael Luckey 
>>> wrote:
>>>
 Hi,

 definitely +1 to properly establish a workflow to maintain jira status.
 Naively I d think, the reporter should close as she is the one to confirm
 whether the reported issue is fixed or not. But for obvious reasons that
 will not work here, so - although it puts another burden on committers, you
 are probably right that the committer is the best choice to ensure that the
 ticket gets promoted. Whether it will be resolved or clarified what's still
 to be done.

 Looking into the current state, we seem to have tons of issues whith
 merged PRs, which for anyone trying to find an existing jira issue to work
 on makes it unnecessary difficult to decide whether to look into that or
  not. From my personal experience, it is somehow frustrating going through
 open issues, selecting one and after investing some (or even more) time to
 first understand a problem and then the PR to realise nothing has to be
 done anymore. Or not knowing what's left out and for what reason. But of
 course, this is another issue which we definitely need to invest time into
 - kenn already asked for our support here.

 thx,

 michel

 On Tue, Mar 12, 2019 at 11:30 AM Etienne Chauchot 
 wrote:

> Hi Thomas,
>
> I agree, the committer that merges a PR should close the ticket. And,
> if needed, he could discuss with the author (inside the PR) to assess if
> the PR covers the ticket scope.
>
> This is the rule I apply to myself when I merge a PR (even thought it
> has happened that I forgot to close one or two tickets :) ) .
>
> Etienne
>
>
> Le lundi 11 mars 2019 à 14:17 -0700, Thomas Weise a écrit :
>
> JIRA probably deserves a separate discussion. It is messy.. We also
> have examples of tickets being referenced by users that were not closed,
> although the feature long implemented or issue fixed.
>
> There is no clear ownership in our workflow.
>
> A while ago I proposed in another context to make resolving JIRA part
> of committer duty. I would like to bring this up for discussion again:
>
> https://github.com/apache/beam/pull/7129#discussion_r236405202
>
> Thomas
>
>
> On Mon, Mar 11, 2019 at 1:47 PM Ahmet Altay

Re: Postcommit kiosk dashboard

Very nice!

Two questions though:
- the links on the left should point somewhere?
- where are the beam_PostCommit_[Java|GO]_GradleBuild coming from? Cant
find them on Jenkins...

On Thu, Mar 14, 2019 at 7:20 PM Mikhail Gryzykhin  wrote:

> we already have https://s.apache.org/beam-community-metrics
>
> --Mikhail
>
> Have feedback ?
>
>
> On Thu, Mar 14, 2019 at 11:15 AM Pablo Estrada  wrote:
>
>> Woaahhh very fanc... this is great. Thanks so much. Love it. - I also
>> like the Code Velocity dashboard that you've added.
>>
>> Let's make these more discoverable. How about adding a shortlink?
>> s.apache.org/beam-dash ? : )
>> Best
>> -P.
>>
>> On Thu, Mar 14, 2019 at 10:58 AM Mikhail Gryzykhin 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I've added a kiosk style
>>> 
>>>  post-commit
>>> status dashboard
>>> 
>>> that can help decorate your office space with green and red colors.
>>>
>>> Regards,
>>> --Mikhail
>>>
>>> Have feedback ?
>>>
>>

Re: JIRA hygiene

I believe that there are too many scenarios that we have to cover if we are
to design a generic approach. Common pattern I've seen most times is when
assignee on the ticket, who's usually author of relevant PR, is expected to
either resolve ticket or pass it to the feature owner for verification.

We can have a bot that will check stale assigned tickets and poke
assignees. Can go further and allow bot to unassign tickets if no response
comes and remove "triaged" label. This will always highlight all
non-updated tickets and keep forgotten tickets in available pool. Giving a
hint to pass ownership of ticket to committer (or person who merged PR) can
be a simple answer for contributors who are not sure whether ticket can be
closed.

--Mikhail

Have feedback ?

On Wed, Mar 13, 2019 at 6:00 PM Michael Luckey  wrote:

> Totally agree. The contributor is most likely be the better target. But as
> she is probably less familiar with the process, we might be better of to
> put the responsibility on the committer to kindly ask/discuss with her how
> to proceed with corresponding jira ticket?
>
> On Thu, Mar 14, 2019 at 1:18 AM Ahmet Altay  wrote:
>
>> I agree with defining the workflow for closing JIRAs. Would not
>> contributor be in a better position to close JIRAs or keep it open? It
>> would make sense for the committer to ask about this but I think
>> contributor (presumably the person who is the assignee of the JIRA) could
>> be the responsible party for updating their JIRAs. On the other hand, I
>> understand the argument that committer could do this at the time of merging
>> and fill a gap in the process.
>>
>> On Wed, Mar 13, 2019 at 4:59 PM Michael Luckey 
>> wrote:
>>
>>> Hi,
>>>
>>> definitely +1 to properly establish a workflow to maintain jira status.
>>> Naively I d think, the reporter should close as she is the one to confirm
>>> whether the reported issue is fixed or not. But for obvious reasons that
>>> will not work here, so - although it puts another burden on committers, you
>>> are probably right that the committer is the best choice to ensure that the
>>> ticket gets promoted. Whether it will be resolved or clarified what's still
>>> to be done.
>>>
>>> Looking into the current state, we seem to have tons of issues whith
>>> merged PRs, which for anyone trying to find an existing jira issue to work
>>> on makes it unnecessary difficult to decide whether to look into that or
>>>  not. From my personal experience, it is somehow frustrating going through
>>> open issues, selecting one and after investing some (or even more) time to
>>> first understand a problem and then the PR to realise nothing has to be
>>> done anymore. Or not knowing what's left out and for what reason. But of
>>> course, this is another issue which we definitely need to invest time into
>>> - kenn already asked for our support here.
>>>
>>> thx,
>>>
>>> michel
>>>
>>> On Tue, Mar 12, 2019 at 11:30 AM Etienne Chauchot 
>>> wrote:
>>>
 Hi Thomas,

 I agree, the committer that merges a PR should close the ticket. And,
 if needed, he could discuss with the author (inside the PR) to assess if
 the PR covers the ticket scope.

 This is the rule I apply to myself when I merge a PR (even thought it
 has happened that I forgot to close one or two tickets :) ) .

 Etienne

 Le lundi 11 mars 2019 à 14:17 -0700, Thomas Weise a écrit :

 JIRA probably deserves a separate discussion. It is messy.. We also
 have examples of tickets being referenced by users that were not closed,
 although the feature long implemented or issue fixed.

 There is no clear ownership in our workflow.

 A while ago I proposed in another context to make resolving JIRA part
 of committer duty. I would like to bring this up for discussion again:

 https://github.com/apache/beam/pull/7129#discussion_r236405202

 Thomas

 On Mon, Mar 11, 2019 at 1:47 PM Ahmet Altay  wrote:

 I agree this is a good idea. I used the same technique for 2.11 blog
 post (JIRA release notes -> editorialized list + diffed the dependencies).

 On Mon, Mar 11, 2019 at 1:40 PM Kenneth Knowles 
 wrote:

 That is a good idea. The blog post is probably the main avenue where
 folks will find out about new features or big fixes.

 When I did 2.10.0 I just used the automated Jira release notes and
 pulled out significant things based on my judgment. I would also suggest
 that our Jira hygiene could be significantly improved to make this process
 more effective.

 +1 to improving JIRA notes as well. Often times issues are closed with
 no real comments on what happened, has it been resolved or not. It becomes
 an exercise on reading the linked PRs to figure out what happened.

 Kenn

 On Mon, Mar 11, 2019 at 1:04 PM Thomas Weise  wrote:

 Ahmet,

Python PVR Reference post-commit tests failing

Hi everyone,

We have Python PVR Reference post-commit tests failing for quite some time
now. These are tests for java reference runner.

According to this thread
,
we are deciding what to do with java reference runner and might want to
remove it from code base.

My question is: do we want to a) invest time in fixing python PVR tests, or
b) disable this test and start cleaning up code?

a) Is worth it if we want to invest into java reference runner in the
future.
b) Is worth if we want to invest into Python and forfeit java reference
runner.

Option b) seem more reasonable to me atm, since most people lean towards
going forward with Python reference runner.

Please, share your thoughts.

Regards,
--Mikhail

Have feedback ?

Re: Python sdist broken?

Thanks for that revert!

I d really love to get those proposed changes of #7675 in, as this
'--keep-temp' also causes some troubles during build.

Please let me know if/how I could support here.

michel

On Thu, Mar 14, 2019 at 5:21 PM Ahmet Altay  wrote:

> Sent https://github.com/apache/beam/pull/8059 to revert #7675.
>
> On Thu, Mar 14, 2019 at 8:10 AM Valentyn Tymofieiev 
> wrote:
>
>> This most likely caused by https://github.com/apache/beam/pull/7675. I
>> suggest we revert and roll-forward with a fix. We should also understand
>> why this didn't surface in the precommit tests.
>>
>> On Thu, Mar 14, 2019 at 4:44 AM Michael Luckey 
>> wrote:
>>
>>> Hi,
>>>
>>> while trying to get build working on my vanilla vm, I encounter
>>> strangely failing python sdists tasks. It keeps complaining about being
>>> unable to copy files. Those files are in a deeply recursive structure which
>>> led me to the assumption this might not be intended. This seems to be
>>> caused by merge of [1], not verified though.
>>>
>>> This also happens on our nightly snapshot build [2]. Anyone else having
>>> this problem?
>>>
>>> thx,
>>>
>>> michel
>>>
>>> [1] https://github.com/apache/beam/pull/7675
>>> [2] https://scans.gradle.com/s/rllnbwj4lj4qc
>>>
>>>
>>>
>>

Re: [BEAM-6835] beam-sdks-python-test-suites-tox-py35:sdist FAILED on clean branch

2019-03-14 Thread Ahmet Altay

PR #8059 reverted a change from yesterday that was causing the sdist issue.
Could you try again and see if your problem is resolved?

On Thu, Mar 14, 2019 at 11:23 AM Alex Amato  wrote:

> https://issues.apache.org/jira/browse/BEAM-6835
> Repro:
> ./gradlew lint :beam-sdks-python:docs
>
> I tested this on a branch I just rebased from master, and the same issue
> is occurring. I also recreated the virtual env.
> https://gradle.com/s/lnlmhu2cggreg
>
> I deleted the build directories, but it still generates multiple files and
> fails.
>
>
> ---
> > Task :beam-sdks-python-test-suites-tox-py35:sdist FAILED
>
> FAILURE: Build completed with 2 failures.
>
> 1: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task ':beam-sdks-python:sdist'.
> > Expected directory '/usr/local/google/home/ajamato/go/src/
> github.com/apache/beam/sdks/python/build' to contain exactly one file,
> however, it contains more than one file.
>
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or
> --debug option to get more log output. Run with --scan to get full insights.
>
> ==
>
> 2: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task ':beam-sdks-python-test-suites-tox-py35:sdist'.
> > Expected directory '/usr/local/google/home/ajamato/go/src/
> github.com/apache/beam/sdks/python/test-suites/tox/py35/build' to contain
> exactly one file, however, it contains more than one file.
>
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or
> --debug option to get more log output. Run with --scan to get full insights.
>
> ==
>
> * Get more help at https://help.gradle.org
>
> Deprecated Gradle features were used in this build, making it incompatible
> with Gradle 6.0.
> Use '--warning-mode all' to show the individual deprecation warnings.
> See
> https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
>
> BUILD FAILED in 17s
> 4 actionable tasks: 4 executed
>
> Publishing build scan...
> https://gradle.com/s/lnlmhu2cggreg
>
>

[BEAM-6835] beam-sdks-python-test-suites-tox-py35:sdist FAILED on clean branch

2019-03-14 Thread Alex Amato

https://issues.apache.org/jira/browse/BEAM-6835
Repro:
./gradlew lint :beam-sdks-python:docs

I tested this on a branch I just rebased from master, and the same issue is
occurring. I also recreated the virtual env.
https://gradle.com/s/lnlmhu2cggreg

I deleted the build directories, but it still generates multiple files and
fails.

---
> Task :beam-sdks-python-test-suites-tox-py35:sdist FAILED

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':beam-sdks-python:sdist'.
> Expected directory '/usr/local/google/home/ajamato/go/src/
github.com/apache/beam/sdks/python/build' to contain exactly one file,
however, it contains more than one file.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or
--debug option to get more log output. Run with --scan to get full insights.
==

2: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':beam-sdks-python-test-suites-tox-py35:sdist'.
> Expected directory '/usr/local/google/home/ajamato/go/src/
github.com/apache/beam/sdks/python/test-suites/tox/py35/build' to contain
exactly one file, however, it contains more than one file.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or
--debug option to get more log output. Run with --scan to get full insights.
==

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible
with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 17s
4 actionable tasks: 4 executed

Publishing build scan...
https://gradle.com/s/lnlmhu2cggreg

Re: Postcommit kiosk dashboard

we already have https://s.apache.org/beam-community-metrics

--Mikhail

Have feedback ?


On Thu, Mar 14, 2019 at 11:15 AM Pablo Estrada  wrote:

> Woaahhh very fanc... this is great. Thanks so much. Love it. - I also
> like the Code Velocity dashboard that you've added.
>
> Let's make these more discoverable. How about adding a shortlink?
> s.apache.org/beam-dash ? : )
> Best
> -P.
>
> On Thu, Mar 14, 2019 at 10:58 AM Mikhail Gryzykhin 
> wrote:
>
>> Hi everyone,
>>
>> I've added a kiosk style
>> 
>>  post-commit
>> status dashboard
>> 
>> that can help decorate your office space with green and red colors.
>>
>> Regards,
>> --Mikhail
>>
>> Have feedback ?
>>
>

Re: Postcommit kiosk dashboard

This is great!

"PreCommit_Cron" are actually very important postcommit runs. Can you add
them?

Kenn

On Thu, Mar 14, 2019 at 11:15 AM Pablo Estrada  wrote:

> Woaahhh very fanc... this is great. Thanks so much. Love it. - I also
> like the Code Velocity dashboard that you've added.
>
> Let's make these more discoverable. How about adding a shortlink?
> s.apache.org/beam-dash ? : )
> Best
> -P.
>
> On Thu, Mar 14, 2019 at 10:58 AM Mikhail Gryzykhin 
> wrote:
>
>> Hi everyone,
>>
>> I've added a kiosk style
>> 
>>  post-commit
>> status dashboard
>> 
>> that can help decorate your office space with green and red colors.
>>
>> Regards,
>> --Mikhail
>>
>> Have feedback ?
>>
>

Re: Postcommit kiosk dashboard

2019-03-14 Thread Pablo Estrada

Woaahhh very fanc... this is great. Thanks so much. Love it. - I also
like the Code Velocity dashboard that you've added.

Let's make these more discoverable. How about adding a shortlink?
s.apache.org/beam-dash ? : )
Best
-P.

On Thu, Mar 14, 2019 at 10:58 AM Mikhail Gryzykhin 
wrote:

> Hi everyone,
>
> I've added a kiosk style
> 
>  post-commit
> status dashboard
> 
> that can help decorate your office space with green and red colors.
>
> Regards,
> --Mikhail
>
> Have feedback ?
>

Postcommit kiosk dashboard

Hi everyone,

I've added a kiosk style

post-commit
status dashboard

that can help decorate your office space with green and red colors.

Regards,
--Mikhail

Have feedback ?

Re: Python sdist broken?

2019-03-14 Thread Ahmet Altay

Sent https://github.com/apache/beam/pull/8059 to revert #7675.

On Thu, Mar 14, 2019 at 8:10 AM Valentyn Tymofieiev 
wrote:

> This most likely caused by https://github.com/apache/beam/pull/7675. I
> suggest we revert and roll-forward with a fix. We should also understand
> why this didn't surface in the precommit tests.
>
> On Thu, Mar 14, 2019 at 4:44 AM Michael Luckey 
> wrote:
>
>> Hi,
>>
>> while trying to get build working on my vanilla vm, I encounter strangely
>> failing python sdists tasks. It keeps complaining about being unable to
>> copy files. Those files are in a deeply recursive structure which led me to
>> the assumption this might not be intended. This seems to be caused by merge
>> of [1], not verified though.
>>
>> This also happens on our nightly snapshot build [2]. Anyone else having
>> this problem?
>>
>> thx,
>>
>> michel
>>
>> [1] https://github.com/apache/beam/pull/7675
>> [2] https://scans.gradle.com/s/rllnbwj4lj4qc
>>
>>
>>
>

Re: Python sdist broken?

2019-03-14 Thread Valentyn Tymofieiev

This most likely caused by https://github.com/apache/beam/pull/7675. I
suggest we revert and roll-forward with a fix. We should also understand
why this didn't surface in the precommit tests.

On Thu, Mar 14, 2019 at 4:44 AM Michael Luckey  wrote:

> Hi,
>
> while trying to get build working on my vanilla vm, I encounter strangely
> failing python sdists tasks. It keeps complaining about being unable to
> copy files. Those files are in a deeply recursive structure which led me to
> the assumption this might not be intended. This seems to be caused by merge
> of [1], not verified though.
>
> This also happens on our nightly snapshot build [2]. Anyone else having
> this problem?
>
> thx,
>
> michel
>
> [1] https://github.com/apache/beam/pull/7675
> [2] https://scans.gradle.com/s/rllnbwj4lj4qc
>
>
>

Re: Executing gradlew build

Hi,

+1 for getting a controlled environment. I am a bit scared to rely on
developers configuration to actually build release artefacts. For instance
during my release process testing I used a docker image, only to realise
after quite a few cycles, that default locale was off which leads to java
using ascii file encodings. This is most likely something we do not want to
happen in real live. And the plain fact, that probably every devel uses a
different jdk build makes - apart from other differences -, the release
build not reproducible, no?

On the other hand, I finally managed to find that Jenkins build, which
actually executes a full build [1]. Unfortunately this also seems to be
consistently failing since March, 1. [2]

[1] https://builds.apache.org/job/beam_Release_NightlySnapshot/
[2]
https://builds.apache.org/job/beam_Release_NightlySnapshot/buildTimeTrend

On Mon, Mar 11, 2019 at 9:41 PM Ahmet Altay  wrote:

>
>
> On Mon, Mar 11, 2019 at 7:03 AM Michael Luckey 
> wrote:
>
>>
>>
>> On Mon, Mar 11, 2019 at 3:51 AM Kenneth Knowles  wrote:
>>
>>> I have never actually tried to run a full build recently. It takes a
>>> long time and usually isn't what is needed for a particular change. FWIW I
>>> view Beam at this point as a mini-monorepo, so each directory and target
>>> can be healthy/unhealthy on its own.
>>>
>>
>> Fair Point. Totally agree.
>>
>>
>>>
>>> But it does seem like we should at least know what is unhealthy and why.
>>> Have you been filing Jiras about the failures? Are they predictable? Are
>>> they targets that pass in Jenkins but not in vanilla build? That would mean
>>> our Jenkins environment is too rich and should be slimmed down probably.
>>>
>>> Kenn
>>>
>>
>> Unfortunately those failures are not really predictable. Usually, I start
>> with plain './gradlew build' and keep adding some '-x
>> :beam-sdks-python:testPy2Gcp -x :beam-sdks-python:testPython' until build
>> succeeds. Afterwards it seems to be possible to remove this exclusions step
>> by step, thereby filling the build cache, which on next reruns might have
>> some impact on how tasks are executed.
>>
>> Most of failures are python related. Had not much success getting into
>> those. From time to time I see 'seemingly' similar failures on Jenkins, but
>> tracing on python is more difficult coming from the more java background.
>> Also using Mac I believe to remember that preinstalled python had some
>> issues/differences compared with private installs. Others are those
>> Elasticsearch - which were worked on lately - and ClickHouse tests which
>> seem to be still flaky.
>>
>
>> So I mainly blamed my setup and did not yet have the time to further
>> track those failures down. But
>>
>> As I did use a vanilla system and was not able to get beam to build, i
>> got thinking about
>>
>> 1. The release process
>> The release manager has lot of stuff to take care for, but is also
>> supposed to run a full gradle build on her local machine [1]. Apart from
>> that being a long lasting task, if it keeps failing this puts additional
>> burden on the release manager. So I was wondering, why we can not push that
>> to Jenkins as we do with all these other tests [2]. Here I did not find any
>> existing Job doing such, so wanted to ask for feedback here.
>>
>> If a full build has to be run - and of course it makes some sense on
>> release - I would suggest to get that running on a regular base on Jenkins
>> just to ensure not to be surprised during release. And as a sideeffect
>> enable the release manager to also delegate this to Jenkins to free her
>> time (and dev box).
>>
>
> +1, this will be great. Quite often we end up catching issues right when
> we are doing the release. I would one up this request and suggest a Jenkins
> job running most of the release process as much as possible to avoid last
> minute surprises.
>
> Also, I wonder if we could build our releases in a controlled environment
> (e.g. container), that way the release would be more reproducible and the
> release manager would have to spend less time setting their environment
> (e.g. environment for building python in your case).
>
>
>>
>> Until now I had not yet have the time to investigate, whether all this
>> pre/Post/elseJobs cover all our tests on all modules. Hoped someone else
>> could point to the list of jobs which cover all tests.
>>
>> 2. Those newcomers
>> As a naive newcomer to a project I usually deal with those jars. After
>> problems arose, I might download corresponding sources and try to
>> investigate. First thing I usually do here is a make/build/install to get
>> those private builds in. Also on our contributor site [3] we recommend
>> ensuring to be able to run all tests with 'gradlew check' which is not to
>> far away from full build. Probably no one would expect a full build to fail
>> here, which again makes me think we need an equivalent Job on Jenkins here?
>>
>> Of course it is also fully reasonable to not support this full build in
>>

Python sdist broken?

Hi,

while trying to get build working on my vanilla vm, I encounter strangely
failing python sdists tasks. It keeps complaining about being unable to
copy files. Those files are in a deeply recursive structure which led me to
the assumption this might not be intended. This seems to be caused by merge
of [1], not verified though.

This also happens on our nightly snapshot build [2]. Anyone else having
this problem?

thx,

michel

[1] https://github.com/apache/beam/pull/7675
[2] https://scans.gradle.com/s/rllnbwj4lj4qc

Re: [PROPOSAL] Preparing for Beam 2.12.0 release